Selection Regimes and Selection Errors

Introduction

Choosing innovation projects comes with inherent difficulty. Organizations face not only substantial uncertainty about market and technological outcomes (Criscuolo et al. 2017, Joseph and Gaba 2020) but also the risk of making errors of commission—selecting unsuccessful projects (false positives)—and errors of omission—rejecting eventually successful ones (false negatives) (Csaszar 2012, Joseph and Gaba 2020). Decades of research on organizational structure and innovation have, accordingly, wrestled with one core question: How should firms organize selection processes to minimize these costly errors (Knudsen and Levinthal 2007)?

Building on foundational work by Simon (1955), one prominent research stream uses models to identify the effects of decision-making structures ranging from hierarchies to polyarchies on both types of errors (Sah and Stiglitz 1986, Christensen and Knudsen 2010). In parallel, observational and laboratory studies examine how real organizations structure their selection processes (e.g., Reitzig and Sorenson 2013, Klingebiel and Adner 2015, Levinthal 2017, Criscuolo et al. 2021), revealing that few firms operate at the extremes of purely hierarchical or polyarchical decision making (Csaszar 2012, Eklund and Kapoor 2022). Scholars increasingly recognize that who selects and how they aggregate information can shape which errors predominate (e.g., false positives versus false negatives).

Yet, selection regime outcomes depend not only on how effectively a firm selects from a pool of alternatives but also on the composition of that pool. Recent studies have begun to disentangle how design choices affect both the generation and selection of submitted ideas (Keum and See 2017, Eklund 2022, Park et al. 2024). This challenge is heightened when external actors, such as entrepreneurs, can strategically choose where and when to submit their ideas (Pahnke et al. 2015, Dahlander et al. 2021, Park et al. 2024). If a selection process becomes more stringent, some entrepreneurs with promising ideas may opt out, whereas others might be strategic in choosing which ideas to submit to this organization versus others.

Complicating matters further, most existing research treats adaptation to environmental shifts—such as adding or removing decision layers—as a single-dimension change (Tushman and Anderson 1986). However, organizations often alter multiple factors simultaneously, including both selection regime structure and the evaluation criteria used, which are not always reliable signals of true potential (Csaszar et al. 2023) and whose application may be cognitively taxing for evaluators dealing with different kinds of innovations (Polidoro 2020). Consequently, the effects of bundled adjustments may be difficult to predict and thus produce unintended consequences.

This paper offers a longitudinal lens on how selection regimes evolve inside organizations. We track one accelerator’s shift through three distinct selection regimes—spanning 3,580 project applications with funding from €20,000 to €70,000—and integrate unique qualitative and quantitative data to capture their effects on the set of ideas submitted for evaluation, selection effectiveness, and the selection errors that emerge. We combine this with 126 interviews with selectors and entrepreneurs to gain a deep contextual understanding of the rationale behind both decisions to change selection regimes and evaluation criteria and the selection or rejection of projects for funding.

Our findings demonstrate that even though additional decision layers and stricter track record criteria improved average submission quality, they also increased false-positive decisions. This finding stands net of accounting for (1) differences in the pool of submissions, (2) differences in treatment effects through training and resources provided, (3) learning, and (4) market evolution. By combining qualitative and quantitative data, we explain this finding through two mechanisms: (1) mean reversion in combination with increased emphasis on applicant track record and (2) within-type adverse selection enabled by a more stringent selection process. Specifically, evaluator decisions became overly reliant on entrepreneur past achievements in a setting more random than they assumed, while some entrepreneurs with prior success submitted marginal projects, devoting less effort to them, and newcomers who may have had good ideas were passed over. By pinpointing these mechanisms, we illuminate why structural changes that seem beneficial can inadvertently undermine selection performance. Our study reveals that changing a selection regime is rarely a simple lever pull; instead, it triggers interconnected shifts in structure, criteria, and entrepreneur behavior that jointly affect who applies and which projects get selected.

Theoretical Background: Structure and Bundled Changes

A long line of research has sought to understand the search for and evaluation of alternatives in organizations, particularly under uncertainty. Simon’s (1955) early work on bounded rationality identified two core challenges: knowing the set of alternatives (the search problem) and figuring out their consequences (the evaluation problem). These issues loom large in innovation selection, where organizations must assess potential products, services, or processes and decide which to fund despite uncertain outcomes. Building on Simon’s insights, the behavioral theory of the firm (Cyert and March 1963) introduced the notion that organizational structures shape how alternatives are generated and ultimately chosen (Knudsen and Levinthal 2007). Subsequent models have emphasized decision makers’ fallibility, stemming from bounded rationality and heuristics (March 1991, Levinthal 1997). Accordingly, organizational choices often fail to produce the desired benefits, in part because of time pressure and attention constraints (Cyert and March 1963, March and Olsen 1975).

Decision making under uncertainty is not just about cost-benefit calculations: biases can creep in even when organizations aim for meritocratic ideals (Boudreau et al. 2016, Criscuolo et al. 2017, Li 2017, Christensen et al. 2023, Csaszar et al. 2023, Piezunka and Schilke 2023). Selection decisions often deviate from a project’s inherent merit (Mueller et al. 2012, 2014; Klingebiel and Adner 2015; Levinthal 2017; Sengul et al. 2018; Criscuolo et al. 2021; Bian et al. 2022). Innovator characteristics (Kanze et al. 2018) and situational factors such as timing (Bian et al. 2022) and sequence (Criscuolo et al. 2021) matter. Research has documented how the allocation of decisions across actors within an organization and the degree to which actors make decisions based on the parochial concerns of their local subunit, or the payoff to the broader organization, affect the adaptive capabilities of an organization (e.g., Dosi et al. 2003, Rivkin and Siggelkow 2003, Ethiraj and Levinthal 2004). For instance, Reitzig and Sorenson (2013) find that managers tend to favor projects from their own divisions. Individual-level biases thus aggregate into an organizational problem, highlighting how structure interacts with evaluation criteria to shape selection outcomes.

Sah and Stiglitz (1986) compared two stylized decision-making regimes: a polyarchy, in which an idea is selected if any one of the managers independently evaluating it supports it, and a hierarchy, where only those ideas sequentially approved by multiple managers are selected. This model shows that hierarchical dependence among decision makers increases the likelihood that a project will eventually not be supported, even though its objective quality recommends it, leading to a smaller share of false positives but a greater share of false negatives. By contrast, a polyarchy’s likelihood of funding a low-quality project is greater, leading to a smaller share of false negatives but a greater share of false positives. Several scholars have expanded these insights by using simulations to understand selection decisions within companies (Levinthal 1997, Rivkin and Siggelkow 2003, Ethiraj and Levinthal 2004, Knudsen and Levinthal 2007, Csaszar 2013, Csaszar and Eggers 2013, Böttcher and Klingebiel 2025). Knudsen and Levinthal (2007, p. 41) extend the work on hierarchies and polyarchies by also considering their hybrids. They argue that “organizations that are hierarchical in structure, even if composed of imperfect evaluators, tend to replicate the conservatism of perfect evaluators and become trapped by local peaks. Hybrid forms, comprising a mixture of polyarchy and hierarchy, effectively balance the dual imperatives of exploration and exploitation.” This observation is essential in that few companies use the extreme forms of hierarchy or polyarchy, with most employing hybrid selection regimes.

Conceptual and modeling work has provided valuable insights into how to structure internally to avoid false positives and negatives (Sah and Stiglitz 1986, Knudsen and Levinthal 2007). But real organizations often do more than tweak one variable in isolation; bundled changes can inadvertently shift both the submission pool and the selection lens. This interplay remains under-studied, particularly over time, where organizations continually adjust their selection processes in pursuit of improvement yet may trigger new forms of error.

Large-scale empirical research on false positives and negatives remains rare because tracking accepted and rejected projects and their outcomes is difficult. When studying false positives, researchers typically analyze the performance of selected projects, which are often subject to survivorship bias. False negatives are even more challenging to observe, requiring knowledge of rejected ideas and their subsequent fate. Some progress has been made by examining discontinuities just below and above a threshold between accepted and “just rejected” ventures, revealing performance gains from selection (Kerr et al. 2014, Hallen et al. 2020), or by using matching methods to compare funded and unfunded firms (Pahnke et al. 2015).

Recent studies exemplify how structure influences selection errors. Csaszar (2012) examines stock‐picking among mutual funds, finding that raising the consensus threshold reduces commission errors but increases omission errors, consistent with Sah and Stiglitz (1986). Keum and See (2017) complement this perspective by showing that hierarchy of authority can dampen idea generation but may also reduce selection errors. Eklund (2022) and Eklund and Kapoor (2022) similarly reveal how centralized versus decentralized R&D structures shape project creation, development, and the use of external innovation.

Overall, recent empirical work on selection processes and outcomes has demonstrated consequential differences between organizational practices and common modeling assumptions and has begun to investigate the influence of organizational structure on both the generation and selection of innovation. However, studies primarily focus on comparing static selection regimes or on a single dimension of change, and we know little about how evaluators reason about changes in how they organize and how entrepreneurs submitting ideas for selection respond. If organizations adapt their selection regimes over time and attract different ideas, then it is plausible that selection regime outcomes depend on both selection regime effectiveness and the set of alternatives available for selection. Accounting for differences in the available pool of ideas and treatment effects, we seek to understand how organizations organize to reduce selection errors. Rather than proposing formal hypotheses, we document what caused the selection regimes to change, using both the evaluators’ and entrepreneurs’ perspectives. We then turn to a quantitative analysis comparing selection errors in each regime, accounting for differences in the pool and treatment effects and evaluating alternative explanations. Lastly, we return to our qualitative and quantitative data to investigate the mechanisms at play.

Data and Empirical Approach

Research Context

AppCampus was a three-year accelerator created in May 2012 through a collaboration among Microsoft, Nokia, and Aalto University in Finland. Microsoft and Nokia each initially committed to providing €9 million in funding,¹ and Aalto University agreed to host the accelerator and cover its operating costs. The program was managed by a team of 8–12 people and was supported by several employees at Nokia and Microsoft. Control was exerted via a steering board of two representatives from Microsoft and Nokia, one from Aalto University, and one Finnish entrepreneur who served as an independent board member.

AppCampus initially had an open application process and offered grants of €20,000 to €70,000 and support to developers of applications for the Windows Phone platform. Submissions underwent a stringent selection process emphasizing application novelty and quality, yielding an overall acceptance rate of approximately 10%. Although the grants did not involve any exchange of intellectual property or equity, applications developed through AppCampus would, once released, be available exclusively on the Windows Phone for initially six and later three months.

By the end of the program in May 2015, AppCampus had received more than 4,300 submissions from more than 100 countries and had invested approximately €10 million in more than 400 app ideas.

Data

To capture changes in the accelerator’s selection regime, we use the detailed qualitative data collection performed by one of the coauthors following a grounded theory approach (Glaser and Strauss 1967). This data collection spanned more than two and a half years (December 2012 to June 2015) and comprised 10 waves (taking place roughly every quarter) of semistructured interviews and informal conversations with key informants at AppCampus, Nokia, Microsoft, and Aalto University; application developers funded by AppCampus; and attendees at key events. Overall, we conducted 126 semistructured interviews, which we complemented and triangulated using archival material from both online and physical sources, such as strategy and review documents, memos, and operational reports. The sources of evidence used to guide our insights are summarized in Table 1.

Table 1. Overview of Data Sources

An interview protocol was developed, used, and adapted as interviews progressed. This enhanced data collection flexibility allowed us to probe emergent themes and take advantage of unexpected opportunities (Eisenhardt 1989). We initially identified the people involved in the creation of AppCampus (Head of Aalto Centre for Entrepreneurship (ACE) and AppCampus’s CEO, COO, Head of Screening, and Head of Quality Assurance) and interviewed them about how AppCampus came to be established and the initial organizational design and processes. On subsequent site visits, we extended our interviews to all AppCampus staff, as well as to the employees of Aalto University, Nokia, and Microsoft who became actively involved in AppCampus operations. The interviews began with us asking about the interviewee’s background and how they became involved with AppCampus, before proceeding to cover their roles in the organization, the work that they did, and any changes in the organization and its interactions with developers and other stakeholders since our last interview with them. Once AppCampus began to host periodical two- to four-week in-person camps for invited funded developers (called “AppCademy”), we strove to interview all of the developer teams taking part² about their background, the app ideas that they were working on, the reasons for them applying to AppCampus, and their experience of the program. All interviews were transcribed verbatim, and the transcriptions were validated by both the coauthor who conducted the interviews and a research assistant, ensuring completeness and clarity of the data. One of the coauthors also had informal conversations with every employee of AppCampus and with a substantial number of the application developers regarding relevant events surrounding the Windows Phone platform and personal sentiments about the value, structure, and operation of AppCampus itself. Further, the coauthor attended several key strategic, planning, and operational meetings and a handful of open days and celebrations, including AppCampus’s first birthday event, a “demo day” organized to display applications of AppCampus participants, and four AppCademy events.

We analyzed the perspectives of AppCampus employees and the funded developers and how these evolved over time as selection regimes changed. We focused on selection (why the entrepreneur chose AppCampus, and how AppCampus chose the entrepreneurs) and treatment (what the entrepreneur hoped to get out of it, and what AppCampus hoped to provide to the entrepreneur). Both coauthors reviewed the interview transcripts and grouped quotes relevant to these broad themes into subthemes. Where a subtheme was represented by two or more quotes within a given selection regime, we compiled them into tables of illustrative quotes, both from the perspective of AppCampus (Online Appendix A) and from the perspective of the developers (Online Appendix B).

We employed a full-cycle research approach (Mortensen and Cialdini 2010) by integrating qualitative and quantitative data to comprehensively understand the mechanisms at play. Our qualitative insights guided the formulation of our quantitative analysis, whereas the quantitative results prompted us to revisit the qualitative data, thereby deepening our understanding. This iterative process allowed us to refine and better explain our quantitative findings (see, e.g., Di Stefano and Micheli 2023).

We gathered archival material from online and physical sources to triangulate primary data, including strategy and review documents, memos, and operational reports. These were examined both during and after the fieldwork period. The archival data were used primarily to complement and verify our primary data. Archival data from industry sources were also used to cross-validate the emerging insights.

To quantify how AppCampus’s changing selection regimes affected their decisions, we use data on all application idea submissions received by AppCampus. After removing duplicates and ineligible submissions, we had data on 3,580 investment decisions. Alongside the full text of the submission document describing the app idea and development team, we have a record of the decision made by AppCampus on every submission. For submissions selected for funding, we have data on their performance directly from AppCampus, which we supplement with hand-collected data following the approach described below to ensure that we account for their performance across different app markets in the years after AppCampus ceased to operate. For rejected submissions, we hand-collected performance data from mobile application stores with the help of four research assistants. The coauthors first took a small sample of submissions and attempted to find information about the release and performance of their proposed applications. Based on this experience, we developed a “search protocol” for the research assistants on how to use multiple different sources, including developer team websites; LinkedIn, Twitter, Facebook, and Instagram pages; mobile application stores; and the Wayback Machine internet archive to find whether that particular developer team eventually released apps corresponding to the app description submitted by the developer to AppCampus³ and, if so, how many times the app was downloaded.⁴ The research assistants first received a file of 100 randomly selected submissions to code. When they finished, they submitted their work to the coauthors, one of whom checked a sample of each file for errors in coding. Two research assistants were found to have a high error rate and were replaced with two others. To further guard against coding mistakes, 242 submissions were coded by pairs of different research assistants to triangulate their work and identify disagreements. A disagreement rate below 6% suggested that the research assistants consistently followed the guidance the coauthors gave; one of the coauthors resolved any disagreements in coding.

Qualitative Findings: The Three Selection Regimes

We use insights from our qualitative data collection to examine how AppCampus changed its selection processes over time. To provide a contextual understanding, we begin by describing AppCampus’s prehistory and first year of operation. After describing its origins, we use our data to explain AppCampus’s and the entrepreneurs’ perspectives during the three selection regimes.

How AppCampus Came To Be

For Nokia and Microsoft, AppCampus was, first and foremost, an investment in the creation of high-quality products and services in the form of applications that would complement the Windows Phone platform. The founding of AppCampus also reflected Microsoft’s and Nokia’s understanding that although their brands, in-house processes, and capabilities were well-suited to attracting experienced developers, attracting younger, less experienced application developers to the ecosystem would be challenging because of the legitimacy deficit these incumbents faced as a result of not having already established successful smartphone ecosystems, unlike their rivals Apple and Google. AppCampus was set up at Aalto University as it was considered to be well-positioned to address a target population of developers who had been largely overlooked by competing ecosystems and whose cost-benefit calculations could be most significantly affected by the incentives offered by AppCampus:

I think Finland’s reach into emerging markets was critical to Aalto’s selection as the university partner to AppCampus. While the grants we are giving may be relatively small to a developer in Silicon Valley, even a €20,000 grant can be very significant to a developer from, say, Russia, Romania, or Poland, covering salary equivalent of over 1 man-year. And these are proving to be among the strongest regions both in terms of deal flow and performance once in the program. —Head of Aalto Centre for Entrepreneurship (ACE)

With many submissions received in the first month of operation, processes were created for evaluating them and selecting the most promising for funding. A quality assurance process was developed to ensure that teams selected for funding met all formal requirements for program participation and would be guided through the development of a design document for their application. Once the design document met the AppCampus standard, 30% of the grant funding was released to the team, with the remaining 70% released once the application was approved for launch in the Windows Phone store.

Alongside building developer capabilities during the award and quality assurance processes, AppCampus also periodically hosted two- to four-week-long training camps for developer teams, paying the costs of developer training, travel, and accommodation. These consisted of sessions on application design and development, and also entrepreneurial skills such as pitching application ideas to investors, marketing, and communications, and concluded with pitches to potential investors at demo days or participation in the pitching competition at the Slush tech, design, and start-up conference.

By the end of 2012, AppCampus had received 1,647 submissions, 80 of which were approved for funding. The first funded apps were released to the public via the Windows Phone marketplace at the end of December 2012. The first released application, a game called “Haunted,” performed well, becoming one of the top 10 most downloaded applications in several countries by March 2013, and reaching hundreds of thousands of downloads by June 2013. It also produced substantial revenue for its developers and received good user ratings. Although other funded apps also succeeded, many did not.

We next discuss the three selection regimes used by AppCampus over time. Table 2 summarizes the key differences among these selection regimes (Online Appendix C gives a more detailed overview of how each selection regime was structured).

Table 2. Overview of Differences Between the Selection Regimes

Selection Regime 1: The Investment Board Between May 2012 and April 4, 2013

In the first stage, the organization used an initial screening performed by AppCampus employees to reject submissions that did not meet the scope criteria (e.g., already released on iOS or Android; app not compatible with Windows Phone) and provide some comments on the eligible cases. These were then sent to an investment board of four people—the AppCampus CEO, AppCampus COO, Head of ACE, and an Entrepreneur in Residence at ACE—who discussed each case together and made the decision by voting, with the CEO casting the deciding vote in the case of a tie. The investment board met once a week and made decisions on 10 to 25 eligible submissions in a given meeting. Novelty of the app idea was the key evaluation criterion at this stage, as stated by the Head of Screening: “The first criterion we have is this innovative side, so just to be something new, if it’s done many many times, we won’t select it, so basically, when I see something that is coming out of nowhere, I’m very happy.” The decision makers were concerned about finding hit apps but also realized the difficulty of doing this, and the trade-offs between false positives and false negatives involved, as the Head of ACE stated: “I’m … worried that maybe, through this process we’ll miss a potential winner. Because that’s the big thing, it’s shown over and over again how easy it is to scoff at something that actually has huge potential. […] You know, I think that is how we have to set the screen at a level that we’re able to get a quality, crazy idea in that might just be huge. And you’re willing to take a certain number of things that won’t fly in order to get that one.”

The investment board comprised different people from different areas of expertise. The reason was that by aggregating information from different people, the board would be more likely to capture novel projects that do not fit the mold and stand out. In practice, decisions taken by the investment board members were not independent. Even in situations where one decision maker had a strong view, this often changed as a result of the discussion, an aspect of the process that was missed if investment board members had to conduct evaluations by email because of travel commitments. As ACE’s Entrepreneur in Residence said, “I … voted through email. And, well, it’s unfortunate because then you are looking at it yourself, you lack the discussion part… there are a couple of cases I … voted totally the opposite. But when we had the discussion, I understood something that I had totally missed.” Group decision-making processes of this kind allow more deliberate decision making, where stated evaluations may change the views of other members before the voting is finalized. As such, the hope was that the group could correct errors made by individual decision makers.

From the perspective of developers, those who applied to AppCampus during this selection regime did so primarily because of the funding enabling them to work on developing their idea full time: “I used to work in a software company and I did these apps as a side project. And then AppCampus was like an opportunity for me to switch from part time to work full time on this” (Developer 8). The developers selected for funding in this regime were often relatively inexperienced, as illustrated by Developer 7: “I didn’t have any of the skills I needed to be able to build like the back-end for it and the database and the website and stuff. So when AppCampus came along, later on, I got those skills, and decided to apply.” They were also often focused on developing for Windows Phone first, with releases on other platforms to follow if the Windows Phone launch was successful: “The plan is to release on Windows phone and then if it’s good, if it’s successful, if the feedback is good and we can fix everything in time then we will most probably release the game as early as possible on other platforms” (Developer 1).

For funded teams, the focus of AppCampus staff during this period was primarily on helping them to improve the quality of their applications: “One of the coolest things is that after you have done applications with AppCampus, you know the quality. I mean, you are very good at quality at that point, and you normally don’t reach that level of quality if you are an independent developer because you don’t need to” (Technical team member). This focus was recognized in our interviews with developers, with Developer 3 stating: “[AppCampus team member] really had some good things to say about the game and he came up with a lot of suggestions for improvements.” The developers who came to the in-person AppCademy training camps also found the networking and pitching training valuable (see Online Appendix B).

Shortly after the program began, issues around scaling began to surface. Because of the number of submissions received exceeding expectations, both investment board members and the screening team who presented cases to them were pressed for time and had to manage competing priorities. As a result, it was difficult to focus attention on all projects on which they had to make decisions. The head of the screening team stated: “We need to keep a fresh mind and this is really difficult, especially when people come and disturb you all the time.” As selection regime 1 (SR1) began to take its toll on the attention of the screening team and the investment board members, they decided to change the process.

Selection Regime 2: Delegation Between April 5, 2013, and November 2013

As the number of submissions for funding grew, the investment board decision making could not keep up. It was becoming increasingly difficult for the investment board to meet. As the COO explained: “It was becoming a bottleneck, so [the regime change] was a work-around to … avoid having these bottlenecks strangling the whole … selection process.” To streamline and speed up the process, evaluation decisions were delegated to AppCampus screening employees. The screening employee responsible for evaluating a given submission had decision authority, and their results were presented to the CEO for review and acceptance. Although most submissions were evaluated one after another in the order in which they had been submitted, in cases where the evaluator thought that additional input from other screening team members could be useful, the app submission and the preliminary evaluation could be discussed at weekly “peer review” meetings. As the Entrepreneur in Residence at ACE said, “The screening team now makes a proposal list, which basically [the CEO] reviews, and stamps that, ‘Okay, this is good.’” The evaluation criteria used also evolved to include an explicit consideration of the app idea’s mass market potential: “[Our scope has changed] to Windows Phone innovation but then combined with potential for mass download or mass appeal” (Head of Screening).

In this selection regime, in addition to the funding offered, potentially privileged access to Windows Phone technology was an important reason for applying for some of the developers, for example, Developer 15: “I think technically it’s quite impressive where Windows Phone is in so little time, and another reason for us to be super stoked about working with them and getting on the platform early.” The cohort selected for funding in this selection regime still included many inexperienced developers, with Developer 20’s story being quite typical: “We were just a bunch of students and we worked on the game for five months before we had a playable prototype.” However, almost all of the teams were actively pursuing or considering a release for their apps across multiple platforms.

Funded teams in this selection regime continued to receive AppCampus’s assistance with the app’s design, although, from the perspective of AppCampus technical staff, the general level of technological understanding had increased because of the increasing availability of online materials, leading them to be contacted less but for more difficult problems: “Most [teams] understand Windows Phone well and currently there is enough research and information on the Internet and everywhere so that they don’t really need help on those basic features. Now when they have a problem they are seriously hard problems, they are not something that you can just Google and find the answer for. So I would say that the technical knowledge has risen a lot and the level of the problems also because of that has risen a lot” (Technical team member). A further aspect of postfunding support that became more of a focus in this regime was assistance with marketing the app postrelease, which was appreciated by several interviewed developers, alongside AppCampus’s input when it came to the business aspects of being an app developer. Those taking part in AppCademy continued to point to the networking aspects as being highly valuable.

At the end of June 2013, the evaluation criteria used by the screening team changed to put additional emphasis on applicant track record. Two months later, this was followed by a substantial change in the way that AppCampus presented itself to potential applicants. In a blog post on the AppCampus website on August 28, 2013, it was announced that “it is time to evolve the selection criteria for funding great mobile applications for Windows phone,” with the main change being that “we are putting more focus on the track record of your team or company. If you have already proven that you can create successful mobile apps (on any platform) you are more likely to get our attention. We’ll be very interested in seeing evidence of your achievements (i.e. download numbers, revenue, rankings and ratings, awards). If you are a new company or have little experience in mobile development the bar will be much higher than before. You would really have to blow us away with an exceptionally innovative idea in order to qualify for an award.” The rationale behind this change was stated as follows by the CEO:

What we found out was that the lead time for getting a quality app out in the store for a start-up that is first time round or nearly first time round, is actually surprisingly long […] We are not a volume program. The latest number that has been announced on number of apps in Windows Phone Store is 170,000, I think. So if we push out 300, 400 teams through the three-year program, it doesn’t really nudge the meter, so that’s not why we are here. We are here to create an environment where the developers who have potential can get successful sooner rather than later; that’s the idea. […] we also need to get stuff out of the system [instead of] just, you know, have a lot of people that get helped along the way, but we also need to have real, hard results.

The emphasis on submissions being “Windows Phone first” was also relaxed for apps that were already successful on other platforms, as long as they would incorporate novel content or features exclusive to Windows Phone. Although not affecting the structure of the selection process at the time, this change demonstrated a shifting of AppCampus priorities from its initial focus on app idea innovativeness as a key evaluation criterion toward a much greater focus on developer track record as a key criterion, in the hope that this would help with delivering high-performing mobile apps.

Selection Regime 3: Preselection Between December 2013 and January 2015

Despite the changes in evaluation criteria, toward the end of SR2, there were growing concerns that the selection process also needed to be reorganized to attract better projects. By having an open submission portal on the AppCampus website flooding the selection process with many app ideas, many lower-quality projects usurped attention. As a result, the selection process was reorganized to match the stated focus on top developers. As the screening team head explained: “It’s not anymore about the … young team that starts and we help the young team getting into the platform. It’s more about the team which is successful and we help them, or we push them to … the next success […] So basically you need to have a successful app to be able to [make] money.” The reorganizations meant that open submissions through the AppCampus website were closed (apart from limited periods coinciding with industry conferences). Instead, submissions were primarily acquired through globally distributed two-day camps each involving approximately 10 developer teams invited by local Microsoft partners, or through direct referrals from Microsoft and other partner organizations. As Microsoft’s liaison to AppCampus explained:

We thought, why don’t we have the local teams … help drive similar activities but in a little bit lighter fashion and do a two-day recruiting event; and they will provide coaching and support. And try to essentially get the teams in … better shape, so that they can submit their ideas to AppCampus and increase the success rate and help us pre-screen those teams.

The top one to three of these teams were put forward for AppCampus funding but still needed to complete an application idea submission, which was evaluated by the screening team in the same manner used in selection regime 2. In using distributed expert opinions (of local Microsoft teams) in this manner, in combination with the emphasis on developer track record as a key selection criterion, the aim was to have a prescreened set of projects for consideration with a tighter quality distribution and a higher mean than the general population. This was echoed in interviews with a screening and tech team member:

Our main channel was this MAAC concept, Mobile Application Acceleration Camp, and that worked extremely well. We had like one per week as an average in the whole spring and we get dozens of new great teams from that channel. So for us it wasn’t- didn’t make that much sense to keep the online open. We would rather try to guide quality teams to the local MAAC events that actually worked pretty okay.

The developers applying to AppCampus in this selection regime tended to focus primarily on the money on offer in explaining their reasons for applying, often planning to use the Windows Phone release as a “soft launch” that enabled them to identify issues to be fixed before launching on iOS or Android: “Because of the funding thing, it’s sort of a chance to get two extra months of development time and to polish the game more, and to see if all our changes work and then get ready for our iOS launch, really” (Developer 24). Compared with the previous selection regime, the resulting cohort of funded developers tended to be more experienced, as illustrated by this quote from Developer 29—“I’ve been through a bunch of programmes similar in nature, not how they’re programmed but in the nature of the things, the subjects that you listen to. I’ve been through four incubators with these and other projects”—and to produce cross-platform apps, with Windows Phone not being their first priority: “[AppCampus] understand the realities that people can’t focus on a platform that is the third most popular, so we have to also think about our business as a whole and also the other platforms” (Developer 45).

In terms of postfunding support, many of the interviewed developers who were selected in this selection regime highlighted the funding as the main benefit. Those who took part in an AppCademy continued to appreciate the networking, pitching, and marketing support. From AppCampus’s perspective, in addition to the continuing marketing support provided to some teams postfunding, members of the Quality Assurance and technical teams became increasingly involved with monitoring the compliance of funded teams in terms of delivering on the features proposed in their applications: “So it might be basically in any stage [of the Quality Assurance process] - me noticing that [developers] try to cut radically their scope, or they cut some key features they have promised to do, for which we have selected them to the programme. So we might say: Hey, we’re not going to pay for that. So either you do what you promised, or you’re out” (Head of Quality Assurance).

Selection Errors by Different Selection Regimes

The qualitative data explain how and why the selection regimes evolved and the entrepreneurs’ perspective of what they hoped to gain from AppCampus. However, they do not give us a comprehensive picture of the selection errors and selection outcomes that these selection regimes produced. To evaluate selection errors in AppCampus’s three selection regimes, we use data on whether submitted app ideas were released in an app store and on the performance of those released, measured in terms of their number of downloads.⁵ We define false positives as app ideas selected for funding that either were not released or were released but generated fewer than 10,000 downloads. False negatives are app ideas rejected by AppCampus that went on to be released and generated either more than 500,000 or more than 1 million downloads, corresponding to AppCampus’s criteria for “successful” and “hit” apps, respectively.

Table 3 shows the descriptive statistics of the number of submissions, projects funded, and selection outcomes in the three different selection regimes. The table provides evidence of the prevalence of both false positives and false negatives across all three selection regimes. There are the fewest false positives in SR2 when considering the percentages of funded unreleased and funded decisions with fewer than 10,000 downloads. We also see that the share of true positives is greatest in SR3, although the numbers are small overall. SR3 also has the lowest share of true negatives. This is to be expected given that this selection was designed to increase the average quality of the pool of submitted ideas. The data on false negatives capture whether rejected projects turned into a success when the team continued to work on them without AppCampus funding. We find that 0.8% of unfunded projects got more than 500,000 downloads. There are differences across selection regimes, with SR3 resulting in the most false negatives (2.5%).⁶ However, the summary statistics above ignore potential differences in the pools of submissions received in the three regimes and other factors that could influence whether an app idea fails or succeeds. Thus, we next turn to using a regression approach to estimate the effect of the different selection regimes on the likelihood of false positives and negatives.

Table 3. Descriptive Statistics—Investment Decisions and Outcomes Across Regimes

Dependent Variables

As discussed above, we consider three dependent variables building on the same categories used by AppCampus in assessing the performance of funded apps. Failure is equal to one if the app idea was not released or was released but generated fewer than 10,000 downloads, and zero otherwise.⁷ Success is equal to one if the app was released and generated more than 500,000 downloads and zero otherwise. Hit equals one if the app was released and generated over one million downloads, and zero otherwise.

Independent Variables

Our key explanatory variables are dummies for each of the three selection regimes, a dummy for whether the app idea was selected and funded by AppCampus, and the interactions between funded and the selection regime dummies. The differences in the pool of app ideas received across the three selection regimes are reflected in the estimated coefficients on the dummies for selection regimes 2 and 3 when their interactions with funded are not included in the models (the dummy for the first selection regime is omitted as the base category).

With the interactions between the selection regime dummies and funded included in the models, the estimated coefficients of the selection regime dummies become estimates of how likely nonfunded app ideas were to fail (true negatives) or become successes or hits (false negatives) in SR2 and SR3, relative to SR1. The estimated coefficient on funded can be interpreted as the association between an app idea being selected for funding using the first selection regime and the likelihood of this idea becoming a failure, success, or hit, whereas the interaction terms between funded and SR2 and SR3 reflect the differences in the likelihood of these regimes producing false positives or true positives, relative to SR1.

Control Variables

We control for team size by counting the number of team members mentioned in the submission form, and for the % female team members by using the genderizeR R package on the first names of team members. We control for gender as previous research has suggested that female entrepreneurs may face biases against them when seeking funding (Brooks et al. 2014, Botelho and Abraham 2017). There are also potential differences in developer experience that accumulate over time and may be correlated with both the likelihood of being selected for funding and producing a hit app. We thus control for whether an app is a sequel to an existing one. We also add a control for whether the app has been ported from a different platform as this may affect uncertainty about the app’s likely performance and could vary among the regimes. Credible signals received about an app idea could also affect the probability of making correct decisions. We thus control for whether the app has a referral code from a different accelerator or laboratory. We include app category fixed effects to account for the possibility that some of the 17 categories of apps that developers classified their submissions into in the AppCampus application form may be more likely to be hits and failures than others.

We also use evaluator fixed effects to account for some evaluators potentially being inherently stricter than others and evaluator experience, measured as the number of submissions that the focal submission’s evaluator has previously reviewed, to control for changes in evaluator judgment that may occur with accumulated experience. At the organizational level, experience with the three different selection regimes may also affect the likelihood of selection errors, leading us to control for regime experience, measured as the number of decisions taken prior to the focal submission within the same selection regime. AppCampus’s portfolio of already-selected app ideas may also have affected decision making and app prospects, so we control for the number of app ideas that AppCampus had already funded in the category of the focal app idea at the time of its submission.

To account for differences in funding among selected ideas, we use a highly funded dummy equal to one if the focal app idea was selected for funding at the €50,000 or €70,000 level (as opposed to the €20,000 granted to over 75% of funded applications), and zero otherwise. Similarly, to account for differences in feedback for rejected ideas, we use a limited feedback dummy equal to one if the focal app idea was rejected and the submitter was informed of this using a generic email containing no feedback specific to the app idea.

Finally, in line with other work that has studied idea evaluation (Piezunka and Dahlander 2019), we also include further controls for textual characteristics of the app idea description submitted to AppCampus in one set of models, as this description was the main means by which app ideas were communicated to evaluators. Specifically, we first included all the dimensions of the text as measured by the Linguistic Inquiry and Word Count (LIWC) text analysis application (Pennebaker et al. 2015). We then proceeded to remove dimensions that appeared to have no association with any of our three outcome variables in a step-by-step process, starting with removing the variable with the highest p-value and proceeding until we were left with only those LIWC dimensions with a p-value of 0.10 or lower in at least one of the models. Our LIWC control variables thus capture the word count of the app description, and the clout, positive emotion, anger, sadness, swear words, and nonfluencies LIWC dimensions.⁸

Estimation Technique

For estimation, as our three dependent variables are binary, we use logit models with robust standard errors, implemented using the logit command in Stata.⁹

Quantitative Results

Table 4 presents the summary statistics of the variables used in the quantitative analysis. Their correlations are presented in Online Appendix G. The estimation results from these models are presented in Tables 5, 6, and 7. Each of these tables has two sets of three columns: the first set of three columns presents results for models excluding interactions between selection regime dummies and funded, whereas the second set of three columns presents results with these interactions included. Within each set of three columns, the first one shows results from models without any control variables or fixed effects for evaluators and app categories, and the second column adds these fixed effects and all non-LIWC control variables, whereas the third column adds the LIWC control variables described above.

Table 4. Regression Analysis Summary Statistics

Table 5. Results—Failures

Table 6. Results—Successes

Table 7. Results—Hits

In Tables 5 to 7, it is important to note the main effects for SR2 and SR3, presented in the first three columns of each table. The results show that SR2 and SR3 resulted in submissions with a lower likelihood of failure (average marginal effects of SR2 and SR3 estimated from a specification including all control variables suggest 10.1% (p = 0.024) and 16.5% (p = 0.019) lower likelihoods of failure than SR1, respectively) and higher likelihood of becoming a hit compared with the first selection regime (5.8% (p = 0.110) and 10.8% (p = 0.059) higher than SR1, respectively). There is also some evidence for submissions received in SR3 being more likely to become successes (although the p-value for the estimated marginal effect of 7.3% is above the conventional cutoff at 0.214). This suggests that the pool of submissions received in the first regime substantially differed from those received in the last two. This is to be expected, given the intentional changes made between the first and third selection regimes to increase the quality of the submission pool through an increasing focus on applicant track record during SR2 and the move to prescreening applications in SR3. The funded variable also works as expected. On average, funded projects are less likely to fail and more likely to result in successes, with limited influence on the likelihood of hits. Specifically, being funded is associated with a 12.4% reduction in the probability of failure (average marginal effect estimated from a specification including all control variables, p < 0.001) and a 2.2% increase in the probability of success (p = 0.005).

To investigate how the selection processes in SR2 and SR3 performed compared with SR1 in terms of the likelihood of selection for funding leading to false positives and false negatives, the variables of interest are the estimates on SR2 and SR3, and their interactions with funded. The estimated coefficients on the funded dummy and its interactions with SR2 and SR3 in the last three columns of Table 5 suggest that SR1 and SR2 were best at avoiding false positives. Specifically, whereas selection for funding in the first two selection regimes is associated with a 13.3% reduction in the probability of failure (p < 0.001), being selected for funding in SR3 was associated with a lower, 7.8%, decrease in the likelihood of failure (sum of the average marginal effects of funded and its interaction with SR3). Table 6 suggests that selection regime 3 was also least effective at identifying true positives. Whereas ideas selected for funding in SR1 and SR2 were 3.8% more likely to succeed than nonfunded ideas (p = 0.002), those selected for funding in SR3 were only 0.4% more likely to succeed.

The estimated coefficients on SR2 and SR3 in Tables 6 and 7 suggest that SR1 and SR2 were best at avoiding false negatives. Specifically, compared with unfunded ideas submitted in SR1 and SR2, unfunded ideas submitted in SR3 were 11.6% more likely to become hits (p = 0.060). Finally, compared with SR1, unfunded ideas submitted in SR2 and SR3 were less likely to fail (by 8.6% (p = 0.058) and 17.3% (p = 0.012), respectively), suggesting that SR1 was most effective at selecting out true negatives.¹⁰

There are, of course, other factors affecting the likelihood of app idea failure or success that our controls seek to account for. As expected, an app ported from a different platform is less likely to fail and more likely to succeed or become a hit. Similarly, sequels are less likely to fail and more likely to succeed or become hits. However, our findings on the selection regimes are robust when controlling for these factors. Another plausible explanation is competition for attention among consumers or different resources available in the category (see, e.g., Ketkar and Workiewicz 2022); that is, there could be more competition from other apps in the same category, both within the accelerator and in the market, in different selection regimes that could explain our results. The selection regime results hold even when including app category fixed effects and the previous number of apps funded by AppCampus in the focal app idea’s category (with the latter having a marginally significant negative effect on the likelihood of the focal app idea becoming a hit). Of the remaining control variables, team size and referral are associated with a lower likelihood of failure but not with becoming successful or a hit, whereas greater regime experience is associated with a lower likelihood of failure and a higher likelihood of hits. Evaluator experience, on the other hand, is associated with a higher likelihood of app idea failure, as is an app being highly funded. Higher measures of the positive emotion and sadness LIWC dimensions seem correlated with a lower likelihood of failure and a higher likelihood of successes and hits. Higher scores on the clout dimension are associated with a lower likelihood of success or the app idea becoming a hit, whereas higher scores on the anger dimension are associated with a lower likelihood of failure. Greater use of swear words is marginally associated with a greater likelihood of failure, whereas greater resort to nonfluencies in the app idea description is associated with a lower likelihood of the idea becoming a hit.

Alternative Explanations

We address four plausible alternative explanations besides changes in selection regime that may explain our findings: (1) changes in treatment effects, (2) changes in the submission pool, (3) learning, and (4) market evolution. These are discussed in detail in Online Appendix H and summarized in Table 8.

Table 8. Overview of Alternative Explanations and How They Were Addressed

Underlying Mechanisms

Now that we go more for quality than quantity, it’s so obvious that the long tail is there still, you can’t escape the long tail. If I were to write a paper on that specific item, the heading would be: You can’t escape the long tail. —AppCampus CEO

The results so far show that SR3 resulted in a greater likelihood of selection errors being made, despite the submission pool in this regime being better than in the first two selection regimes. These results are robust when controlling for alternative explanations, innate differences between evaluators and app categories, and language characteristics of the description of the project as submitted by its developers. Two plausible mechanisms can account for why SR3 had worse results, given the pool of available submissions: (1) mean reversion in combination with an increased emphasis on applicant track record and (2) adverse selection enabled by a more stringent selection process. We elaborate on each of these below and consider the extent to which they resulted from changes in evaluation criteria during SR2 versus the additional screening steps introduced in SR3.¹¹

Mean Reversion in Combination with Increased Emphasis on Applicant Track Record.

The change in evaluation criteria during SR2, which increased the emphasis on applicant track record, in combination with the prescreening approach introduced in SR3 set the bar high for submissions to be introduced into the consideration set with the aim of tightening the quality distribution. Several months into SR2, the AppCampus screening team began to rely on developer team track record as a key evaluation criterion and weeded out many teams without one, attributing the prior success to teams and their capabilities: “The bar is higher. We are choosing, in a way, safer bets… If somebody has already proven that they can make a successful mobile app, I think it’s easier for us to believe that they would make another one. And that’s most of the cases” (Screening team member). This rationale was echoed by many other members of AppCampus, including the CEO: “[Developers with track records] have already done something, and they might have found the recipe how to become successful.” Using the previous record of accomplishment can lead to benefits if it is informative about the probability of a follow-on success. But teams that were successful once may have difficulty following that up, and when this escalates to over-reliance on previous records of accomplishment, problems can occur.

To evaluate the extent to which our results are driven by the changes in evaluation criteria during SR2 versus the change in the selection process from SR2 to SR3, we looked through the evaluation forms completed by the AppCampus screening team and identified how these changed over time. Specifically, the initial evaluation criteria used throughout SR1 and the first part of SR2 were whether the app was “innovative/first to market,” “differentiated/not available on other platforms,” “supported key Windows Phone features,” “demonstrated design elegance and technical quality,” and “had potential to drive ecosystem momentum through device sales or high user numbers.” On June 27, 2013, these criteria changed to the extent to which the submission showed “differentiation/innovation,” “exclusive features,” “mass-market appeal,” “developer track record,” and a “convincing proposal for app development and user acquisition.” As this change may have influenced the selection of ideas and the resulting selection errors both in the latter part of SR2 and in SR3, regardless of any changes in the structure of the selection process, the first three columns of Table 9 present the results of fully specified models but with the variables capturing changes in selection regimes and their interactions with funded replaced by a new evaluation template dummy variable equal to one for ideas submitted on or after June 27, 2013, and zero otherwise, and its interaction with funded. As can be seen in columns 1–3 of Table 9, making these changes produces results suggesting no difference in the likelihood of nonfunded submissions before or after them becoming failures, successes, or hits. For funded submissions, there is a marginally significant difference (p = 0.058) in their likelihood of failure after the change to the new evaluation template, with no differences in the likelihood of funded app idea success or becoming a hit. Thus, although the change in evaluation criteria does seem to have somewhat contributed to the greater likelihood of false positives in SR3, it does not appear, by itself, to explain the greater likelihood of false negatives nor the lower likelihood of true positives in the results in Tables 5–7.¹² Although the strengthening of evaluation criteria would by itself be expected to result in a greater share of false negatives, even if the selection regime was otherwise unchanged, the results in Tables 5–7 and 9 show it only seemed to make a substantial difference once it was reinforced by the change to SR3 extending the number of times that a submission had to clear the “track record” bar to be funded.

Table 9. The Effects of Changing Evaluation Criteria

Our empirical work highlights Harrison and March’s (1984) observation, where evaluators may select the project with the highest estimate. However, because each estimate has a noise component, selecting the best tends to be associated with more postdecision surprises. Prior success may be over-attributed to the team’s capabilities rather than the luck and good fortune that helped them in the process (see also Denrell and Liu 2012, 2021). With regression toward the mean, this then leads to lower-than-expected outcomes. By focusing too much on past track record, the screening team may have failed to appreciate that the developer’s next project may not be as successful as their earlier success. Furthermore, ignoring submissions by teams lacking such a track record, regardless of the app idea’s other qualities, risked missing future hits. Looking through the comments on false negatives—rejected opportunities that later had more than one million downloads—we do find some support for this argument. Of the twelve false negatives in SR3, a common reason given for rejection was “No track record given.”

To further investigate whether there is evidence of this mechanism in our data, we examined the reasons for rejecting each of the 26 false-negative cases in our data set. These reasons across the three selection regimes are summarized in Table 10. In SR1, two of the six false-negative cases were proposed for funding by the investment committee as exceptions to the “no porting cases” policy in force at the time but were rejected by the steering board, with “lack of innovation” and “execution barriers” being the next two most common reasons. In SR2, “lack of innovation” was the most common reason for rejection, followed by “lack of mobile development skills.” In SR3, the most common reason for rejection was “lack of track record,” followed by “borderline cases” in which the evaluation comments, though positive on the whole, stated that they were not fully convinced by the submission, leading to eventual rejection.

Table 10. False-Negative Rejection Reasons

The patterns of rejection reasons for submissions that became successful apps are consistent with the qualitative evidence and mean reversion arguments: false-negative cases shifted from a nearly exclusive focus on evaluating the strengths and weaknesses of the proposed app idea in SR1, to most of these cases being rejected because of the team’s lack of track record in SR3.

Within-Type Adverse Selection Enabled by a More Stringent Selection Process.

SR3 set the bar high for submissions to tighten the quality distribution because of a combination of higher emphasis on developer track record and greater hierarchy in the selection process. However, developers with established track records are likely to have developed their craft on competing Android and iOS app stores, which had much larger user install bases because of their earlier entry. Top developers targeted by AppCampus in this regime would also likely have access to alternative funding sources for their app ideas, whether from their earnings or other investors. Once the increased focus on track record as a key selection criterion was broadcast in the AppCampus blog post on August 28, 2013, if such developers had a better understanding of the quality of their app ideas compared with AppCampus evaluators, they could potentially have exploited their information advantage to use AppCampus for funding app ideas they considered to be lower quality compared with other app projects that they had in development, and so less likely to be funded by other means. In other words, they may not have submitted their best projects to AppCampus. The potential for this would have further increased with the change to the SR3 selection process resulting in the track record criterion being evaluated multiple times by different evaluators. AppCampus’s reliance on the team’s track record as a key criterion in this regime could thus have backfired because of within-type adverse selection (see, e.g., Huangfu and Liu 2023, Nguyen and Tan 2023), further explaining the regime’s lower performance in avoiding false positives and selecting true positives. This concern was raised by a member of the AppCampus screening team: “The guys with the brightest ideas and access to more resources and with more entrepreneurial experience, they may already have the iPhone App or the Android App, so it’s a bit difficult to get these guys to apply for the program or then, if they apply, then maybe they are already in a stage where they don’t really need us anymore.”

As the potential mechanism sketched out above requires the submitters with multiple funding options to know that their submission is likely to be viewed more favorably by the focal organization, we first investigate the effects of the explicit announcement of the increased weight given to developer track record in the blog post on the AppCampus website on August 28, 2013. Columns 4–6 of Table 9 present the results of fully specified models but with the variables capturing changes in selection regimes and their interactions with funded replaced by a post-blog post dummy variable equal to one for ideas submitted on or after August 28, 2013, and zero otherwise, and its interaction with funded. These results show that nonfunded submissions after this change were marginally less likely to fail (p = 0.057), and the effect of funding on the likelihood of funded app failure was similar, though weaker, to the results for SR3 and its interaction with funding presented in Table 5. However, this change had no effect on the likelihood of nonfunded submissions succeeding or becoming a hit, nor on the likelihood of funded submissions achieving these outcomes. Overall, similar to the results for the changes in evaluation criteria presented in columns 1–3 of Table 9, this announcement of the changed evaluation criteria does seem to have contributed to the increased likelihood of false positives observed in SR3 but does not seem to explain SR3’s higher likelihood of false negatives nor its lower likelihood of true positives.¹³

To further investigate evidence of the adverse selection mechanism in our data, we examined each of the 254 false-positive cases. Using the data on the progress of each idea from submission to either release or termination, we were able to classify false-positive cases as having failed for one of three reasons: achieving fewer than 10,000 downloads postrelease; being terminated after the app idea had passed the design milestone and the developer had been paid 30% of the total grant amount; and being terminated before the app idea had passed the design milestone, without the developer having been paid any of the grant amount. The latter two categories contain cases that were terminated because of the app idea not meeting the quality standards to pass the next required milestone (release candidate and design milestones, respectively), those that were terminated at the developer’s request, and those that were terminated because the developer had stopped responding to communications from AppCampus. These data allowed us to see the extent of interaction between the AppCampus teams and the application developers throughout the process, as well as the share of false-positive cases that received additional support from AppCampus, either through participation at the AppCademy residential training camps or through the provision of marketing support, mostly in the form of free impressions on an app advertising network.

The resulting summary statistics for each selection regime are presented in Table 11. The share of false positives because of low number of downloads increased from 48.9% to 59.6% between SR1 and SR3, whereas SR2 saw the highest share of false positives occurring without the app in question being released, but after the developer had received 30% of the grant amount (29.4%). SR1 saw the highest share of false positives because of projects being terminated prior to passing the design milestone, without the developer receiving any of the grant amount (27.7%). Although these statistics are not proof of adverse selection becoming more of an issue in SR3, they are consistent with that possibility as substantially more failed app projects selected for funding in this regime failed not because of their developer’s inability to deliver the level of design and technical quality expected, but rather because of poor postrelease performance.¹⁴

Table 11. False-Positive Failure Reasons and Summary Statistics

Further examination of the comments from the AppCampus team on each false-positive case provided additional supporting evidence for the adverse selection mechanism. Although none of the comments on cases in SR1 mentioned aspects of the team’s performance consistent with adverse selection, there was one such case in SR2 (“The title is too good to be doing so poorly! Part of the issue is price, but I also think they just haven’t done much to promote this title either. Need to get them more engaged”), and six such cases in SR3 (e.g., “They are working on the version 3 already and are not interested on the version 2 anymore”; “In terms of studio priorities [they] have moved on from the project under discussion. [They] are working on other games now”; “Team’s commitment seems a bit low”; “The team has been approached by a number of publishers over the past couple of months and as a result of those conversations have decided that they cannot bring this title to Windows Phone first”).

As in the mean reversion mechanism above, a combination of changing evaluation criteria and the additional layers of screening introduced in SR3 seems to produce this mechanism. Empirically, the results presented in Tables 5–7 and 9 suggest that although the change in selection criteria drives the greater share of false positives in SR3, the lower performance of SR3 in the selection of true positives is observed only once these changes are compounded by increased hierarchy in the selection process.

Discussion

Rich conceptual, modeling, and experimental work has considered how to organize for selecting innovation projects (Csaszar and Eggers 2013), yet we know little about how organizations adapt to improve their selection and the effect of this on selection errors (see Keum and See (2017) and Eklund (2022) for exceptions). Our qualitative work documents how selection regimes evolved and adapted in structure, the evaluation criteria used, and the selectors’ and entrepreneurs’ divergent perspectives of the selection regimes. Despite changes to enhance selection regime performance, our quantitative findings reveal similarity in overall selection errors across these regimes. However, this similarity conceals contrasting effects of the changes made on the pool of submitted ideas and the effectiveness of selection among these ideas. Although eliciting a higher-quality pool of submitted ideas, the third and most selective regime was less effective in avoiding false positives and negatives than its predecessors. By combining our qualitative and quantitative data, our findings suggest two mechanisms that explain this: (1) mean reversion combined with an increased emphasis on applicant track record and (2) adverse selection caused by a more stringent selection process, which led some experienced teams to use the accelerator for projects where they did not fully commit. We elaborate on the theoretical implications for evaluation and selection literatures below. Finally, we discuss how using thick qualitative and quantitative data from a single accelerator may limit the generalizability of our findings, which future research could address.

Theoretical Implications

Most empirical studies of selection study a single regime (see, e.g., Criscuolo et al. 2021) or compare regimes across organizations (e.g., Csaszar 2012). This study contributes to the literature by highlighting adaptation in selection processes and demonstrating that organizations respond to emerging challenges by making bundled changes to their selection regimes. This underscores flexible organizational design, where changes are made in response to real-time feedback and evolving circumstances rather than through a static, predetermined approach. We find adaptation within organizational selection processes without evidence of effective learning, possibly because feedback from the market on what works can take a long time to manifest. It is important to remember that the selection regimes we examine do not precisely align with those in existing studies (see Sah and Stiglitz 1986, Knudsen and Levinthal 2007, Csaszar 2012). Specifically, AppCampus incorporated specialists with overlapping yet complementary expertise to leverage diverse experiences and viewpoints and used multiple criteria for evaluating which projects to fund. Additionally, well-intended changes to selection regimes to address emerging issues may be bundled, making it more challenging to foresee outcomes and leading to unintended consequences.

Our findings suggest selection regime changes can affect both the pool of ideas submitted to the organization and the regime’s performance in selecting among them, which has implications for work on selection that has treated the pool of ideas as fixed. For instance, Sah and Stiglitz’s (1986) predictions about greater hierarchy in the selection process being associated with more true positives hold in our data, as they do in Csaszar (2012), but not because the SR3 selection process is better than the prior, less hierarchical ones, at finding true positives. Instead, this more hierarchical process resulted in a greater likelihood of true-positive outcomes because of a higher-quality set of ideas self-selecting into it despite being less effective at selecting among these ideas. Our results also complement and contrast with the findings of Keum and See (2017). Although that paper hypothesizes and finds increased hierarchy of authority having a detrimental effect on idea generation, primarily in terms of the number of alternatives generated by participants in a laboratory experiment, we find that increased selection process hierarchy in SR3 has a positive effect on the pool of ideas submitted for evaluation in terms of their likelihood of avoiding failure and being successful. Our work thus provides a complementary perspective on the effects of the selection process hierarchy on the pool of alternatives when these come from outside the focal organization, where the mechanism of self-censoring to conform to supervisor preferences proposed in Keum and See (2017) is likely less relevant. Similarly, our findings of the effects of hierarchy on selection among alternatives contrast with those in Keum and See (2017), likely because the reduction of bias in the evaluation of own ideas that drives their findings is ruled out by design in the organization we study, as nobody involved in AppCampus operations could submit app ideas for evaluation. Our findings therefore contribute to prior work on selection by helping us to understand better the likely effects of selection regimes both on the pool of alternatives submitted to the organization and on errors in selecting between these alternatives in settings where innovation ideas may come from outside of the organization (e.g., Pahnke et al. 2015, Lifshitz-Assaf et al. 2021).

Considering both changes in evaluation criteria and the decision-making structure over time, we find evidence of two novel mechanisms that appear to make selection among the higher-quality set of alternatives submitted in SR3 more challenging. The first is a variant of mean reversion of performance in combination with an increased emphasis on applicant track record. By separating innate attributes of projects and the associated applicants rather than considering the quality of the project as a unidimensional measure (Knudsen and Levinthal 2007), we can distinguish how evaluation can have unintended consequences. When evaluators placed heavy emphasis on teams’ prior track record, the quality of the idea itself lost some of its significance. This parallels how venture capitalists (VCs) often hail the team as more important than the idea. The flip side of this argument is that if the process is more random than assumed (Denrell and Liu 2012), then past team track records may be less informative about future performance and thus given too much weight in the decision.¹⁵ The second mechanism of within-type adverse selection from SR3 targeting teams who likely have multiple outside options for funding their best ideas and who potentially use AppCampus to fund their inferior ideas also has important implications. Recent work on within-type adverse selection (e.g., Huangfu and Liu 2023, Nguyen and Tan 2023) considers ways information availability across markets and bundled trades may reduce its detrimental effects. These potential solutions may be more or less viable based on how the organization soliciting and evaluating ideas is designed. In the case of AppCampus, the combination of their focus on selecting among particular app ideas and the grant-based support provided precluded using a bundled trade approach. However, accelerators that provide their services in exchange for an equity stake in the firms that they support effectively benefit from the success of all of the selected firms’ current and future projects and so may be able to reduce the likelihood of such within-type adverse selection. Even if the context we study is one in which innovation projects are submitted from outside of the selecting organization, both of the above mechanisms affect within-firm selection regimes, as evaluation criteria may not always reflect the best indicators of desired outcomes and because employees may have options for finding greater support and rewards for their best ideas elsewhere. The extents to which the above mechanisms are affected by the openness of the organization’s innovation process and the relative standing of the organization (and its ecosystem) are interesting questions for future research.

Our findings can be seen in the light of previous empirical work on the value of accelerators (see, e.g., Hallen et al. 2020, 2023) and how they organize (Cohen et al. 2019). For example, Hallen et al. (2020) use inverse probability treatment weights to estimate the effects of top accelerators and find mixed effects for their impacts on venture development. Similarly, Yu (2020) examines how accelerator participation impacts venture performance by leveraging quasi-experimental methods, finding that accelerators provide feedback that helps founders resolve uncertainty about their ideas more quickly, leading to earlier and more frequent closures of ventures, as well as reduced funding amounts for those that do close. In our setting, by contrast, the effects of changing selection regimes come from a combination of changes in the effectiveness of the selection process and in the pool of alternatives available for selection rather than changes in the value that the accelerator brought to the funded teams. In a related paper, Cohen et al. (2019) find huge variation across accelerators in their organizational design. Using unique within-accelerator data, we show that there may also be major variation in design within an accelerator over time and that such changes are likely to affect both the entrepreneurs applying and the effectiveness of selecting among them. Although the changes that we observe generally proceeded in the direction of greater selectiveness and more hierarchy in the selection process, it is not obvious whether we would expect opposite findings if the progression between selection regimes had been reversed, as starting with a more hierarchical selection regime with strict evaluation criteria may establish a harder-to-change perception among potential applicants of what the accelerator is looking for, potentially reducing the variety that organizations later get exposed to (Park et al. 2024). Innovation results from considering many different options, meaning most ideas fail or are rejected on the journey toward adoption. Analyzing all projects shows that some rejected ideas succeeded. Of the 32 submitted app ideas that achieved more than one million downloads, 16 were funded by AppCampus. In every selection regime, the best-performing unfunded (false-negative) submission generated downloads equal to at least 40% of the total downloads generated by all funded submissions in that regime. This highlights the difficulty of separating “good” from “bad” projects and the magnitude of potential consequences. This challenge is consistent with Yu’s (2020) emphasis on the conditional and context-dependent nature of accelerator outcomes, as accelerators must navigate inherent uncertainty when selecting ventures to support.

Limitations and Future Research

Our paper has several limitations, most notably that of studying a single accelerator. Little research exists on how rejected ideas lead to commercial success for many projects in other settings, so it becomes difficult to know how well the findings generalize. AppCampus used different ways to organize and changed multiple aspects of its selection regimes. Despite ruling out several alternative explanations in our post hoc tests, several limitations remain and need to be addressed in future work.

First, a limitation of our study is the presence of simultaneous events in the environment that could confound our results. Although we have attempted to mitigate these confounding influences by incorporating fixed effects for different app categories, controlling for the performance of the Windows Phone ecosystem over time, and examining the interplay between selection regimes and treatment effects, these measures may not fully capture the complex dynamics at play. The broader external environment, including market fluctuations, technological advancements, regulatory changes, and competitive actions, can significantly impact the availability and nature of projects and the effectiveness of selection regimes. Future research could leverage exogenous shocks in the external environment to address this limitation. Such shocks could include sudden regulatory changes, significant technological breakthroughs, economic crises, or shifts in consumer preferences, which are external to the organization and thus provide a source of variation independent of the internal processes under study. Analyzing how organizations adapt their selection regimes in response to these natural experiments could yield valuable insights into the robustness and adaptability of different selection strategies under varying conditions.

Second, a difficulty with considering false negatives is that rejected ideas may pivot because of feedback received, which makes it more difficult to trace such ideas (Tidhar and Eisenhardt 2020). We painstakingly coded the fate of all rejected projects, but we might have missed projects that transformed so much that we could no longer trace them. If these could still be considered the same ideas as those evaluated by AppCampus, we would underestimate the true number of false negatives. This is difficult to rule out completely, but we would not expect this to differ systematically across the selection regimes.

Third, an inherent limitation of our study is that the modifications to the selection regimes were not exogenous but rather emerged from the organization’s learning and adaptation processes. This characteristic inherently restricts our ability to draw strong causal inferences about the direct impact of these changes on organizational outcomes. Recognizing this limitation, future research would benefit from adopting an experimental approach. Such an approach could more precisely disentangle the separate and joint effects of changing evaluation criteria and selection regime structure on both selection effectiveness and outcomes and test our proposed causal mechanisms. Furthermore, extending the analysis to include a comparative study of various organizations with differing selection regimes would provide valuable empirical evidence to support or challenge our findings. Such comparative analyses, potentially leveraging longitudinal data, would enrich our understanding of how different organizational cultures, structures, and strategies influence the efficacy of selection regimes. It is likely, however, that convincing organizations to run field experiments with different selection regimes will be challenging.

Conclusion

Managing idea selection and preventing false positives and negatives is one of the most critical organizational processes affecting both performance and employee morale (Giarratana et al. 2018). Our study suggests that mistakes happen frequently and highlights the potentially contrasting effects that changing selection regimes can have on the pool of ideas submitted for evaluation and the challenges of selecting among these ideas. We hope our findings help managers design and implement more effective selection processes and inspire future research to further expand our understanding of organizational selection processes and their intended and unintended effects.