Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality

1. Introduction

The capabilities of artificial intelligence (AI) systems to aid humans in a variety of tasks have improved rapidly, especially since the release of OpenAI’s ChatGPT, which made large language models (LLMs) widely available for public use. Recent research has shown that these systems are unexpectedly capable of augmenting human abilities in the completion of creative, analytical, and writing tasks as well as achieving top scores in academic aptitude tests at graduate and professional levels (Geerling et al. 2023, Noy and Zhang 2023, Boussioux et al. 2024, Brynjolfsson et al. 2025). This represents a significant shift in human-machine augmentation, with hybrid aptitudes increasingly relevant across a broad range of knowledge work—including some highly creative and highly educated professions (Eloundou et al. 2024, Zhou and Lee 2024).

As generative AI’s capabilities increasingly overlap with human skill sets, the integration of AI into human work presents new fundamental challenges, opportunities, and questions, especially in knowledge-intensive scientific and professional fields. In this paper, we investigate the impact of generative AI on management consulting—a crossdisciplinary field where top graduates from leading universities tackle diverse projects across industries. We examine this issue using a randomized, controlled laboratory-in-the-field experiment, in which highly skilled management consultants performed tasks resembling their standard work assignments.

Our results demonstrate that AI capabilities cover an uneven set of knowledge work, and we introduce the concept of a “jagged technological frontier” to characterize the performance of AI when used by knowledge workers. Tasks that appear to human knowledge workers to be of similar difficulty may be performed either better or worse by humans using AI. Within this jagged frontier, AI can complement human work. However, outside the frontier, AI output is inaccurate, is less useful, and can degrade human performance. AI assistance improves human performance only for tasks within current AI capabilities—within the jagged technological frontier—and worsens human performance outside of it. We find that workers who skillfully navigate this frontier in their use of AI systems gain substantial quality and productivity benefits.

Figure 1 provides a conceptual representation of this jagged technological frontier, illustrating how tasks that appear similar in complexity for humans may nonetheless fall on opposite sides of the boundary where AI assistance helps versus harms performance. Notably, jaggedness reflects systematic differences in how well AI capabilities align with different tasks compared with expectations based on human capabilities. Because the capabilities of AI are rapidly evolving and poorly understood, it can be hard ex ante for knowledge workers to grasp exactly what the boundary of this frontier might be at a given moment in time. Indeed, within the same knowledge workflow, some tasks are beyond the frontier, whereas others remain within it, making effective AI use challenging—especially without diligent examination of the workflow to identify deviations from expectations and the potential for redesign.

The potential benefits of AI for knowledge work have been a subject of considerable interest, although much of this research focuses on AI prior to the release of GPT-3.5 (e.g., Brynjolfsson et al. 2018, Çalli et al. 2021, Davies et al. 2021, Monisha et al. 2021). The general availability of GPT-3.5 in the form of ChatGPT since November 2022 has changed both the nature and urgency of the discussion (Berg et al. 2023, Eloundou et al. 2024, Zhou and Lee 2024). Studies of previous generations of AI (e.g., Agrawal et al. 2018, Furman and Seamans 2019, Brynjolfsson et al. 2021) and of recently released LLMs (e.g., Noy and Zhang 2023, Choi and Schwarcz 2024, Brynjolfsson et al. 2025) suggest that these systems can considerably impact worker performance. Our study focuses on complex tasks selected and evaluated by industry experts to replicate the real-world task assignments of knowledge workers. Most knowledge work includes a set of interdependent tasks, some of which may be a good fit for current LLMs, whereas others are not. We examine multiple types of interdependent knowledge tasks and build on recent studies to suggest ways of understanding the impact of generative AI on real-world task assignments, under which circumstances knowledge-based organizations may benefit but also face negative outcomes, and we consider how this impact might change as the technology advances.

Three factors make it difficult for knowledge workers to know what the value and downsides of using generative AI may be ahead of time. First, LLMs have surprising capabilities that they were not explicitly or intentionally developed to have—capabilities that are growing rapidly over time as model size, computational capability, and algorithmic performance increase. Although trained as general models, LLMs demonstrate specialist knowledge and abilities as part of their training process and during normal use (Reed et al. 2022, Boiko et al. 2023, Moor et al. 2023, Singhal et al. 2023). Although considerable debate remains on the concept of emergent capabilities from a technological perspective (Schaeffer et al. 2023), the effective capabilities of LLMs and the ways that they are used are novel and unexpected, widely applicable, and increasing rapidly over short time spans. Recent work has argued that AI performs at a high level in knowledge work contexts ranging from medicine to law (Ali et al. 2023, Lee et al. 2023) and that the human-AI combination outperforms humans alone on many measures of innovation (Boussioux et al. 2024, Meincke et al. 2024). Additionally, scores on various standardized academic tests, albeit an imperfect measure of LLM capabilities, have been increasing substantially with each generation of AI models (OpenAI 2023).

The general ability of LLMs to solve domain-specific problems leads to the second factor differentiating LLMs from previous forms of AI: their ability to directly improve the performance of knowledge workers who use these systems and more generally, their unprecedented usefulness in knowledge-intensive domains. Early studies of the new generation of AI suggest direct performance increases from using AI, especially for complex work, like writing tasks (Noy and Zhang 2023) and software programming (Peng et al. 2023), as well as for ideation and creative work (Boussioux et al. 2024, Meincke et al. 2024). Consequently, the effects of AI are expected to be higher in knowledge-intensive domains featuring highly educated workers (Felten et al. 2023, Eloundou et al. 2024).

The final relevant characteristic of generative AI is its relative opacity. Although previous generations of AI are often black boxed, which can be frustrating for users (Lebovitz et al. 2022), LLMs have been shown to produce incorrect but plausible results (hallucinations or confabulations) in a way that users find accessible and “believable,” leading to inaccurate outputs.¹ The advantages of LLMs, although substantial, are frequently unclear to users. Despite performing well at some tasks, LLMs fail in other circumstances in ways that are difficult to predict in advance. Adding to this opacity, developers provide no guidance regarding the best way to use these AI systems, which appear to be best learned via ongoing user trial and error and the exchange of experiences and heuristics on various online forums.

These three factors together suggest that both the value and downsides of using generative AI may be difficult for knowledge workers to know ahead of time. Some unexpected tasks (like idea generation) are easy for AIs, whereas tasks that seem as though they should be easy for machines to complete are challenging for some LLMs (e.g., basic math). Our research suggests that this feature of LLMs as used by humans creates a “jagged frontier,” where tasks that appear to be of similar difficulty are performed better or worse by humans using AI. Because of the “jagged” nature of the frontier, a knowledge worker may face tasks in an assignment that are on both sides of the frontier without a priori knowledge regarding each task’s position with respect to that frontier. Knowledge workers may fail to see AI’s limitations and become overly reliant on AI’s output, potentially overlooking its limitations (Dell’Acqua 2022). The future of understanding how AI impacts work involves understanding how human interaction with AI changes depending on where tasks are placed on this frontier. This work investigates how humans navigate this jagged frontier and the subsequent performance implications.

For this study, we collaborated with a global management consulting firm (Boston Consulting Group (BCG)), advising them on the design, development, and execution of a preregistered randomized experiment to assess the impact of AI on knowledge workers.² We specifically worked with them to develop tasks that closely resemble the workflow of their management consultants. Subsequently, the author team received the data that the company collected for this experiment and conducted the analysis presented in this paper. Online Appendix F shows a detailed analysis using Bureau of Labor Statistics data that contextualizes our experimental occupation within the broader landscape of knowledge work, demonstrating how management consulting responsibilities align with the multifaceted demands of modern professional environments.

We tested two distinct sets of tasks: one situated beyond the frontier of AI capabilities and one situated within the bounds of that frontier. The experiment aimed to understand how AI integration with human-performed tasks might reshape the traditional workflows of highly skilled professionals in a knowledge domain. Both sets of tasks were developed to be realistic and were designed by senior professionals at BCG who had experience in the same roles as the participants during their careers and were currently supervising workers in those roles. A senior managing director and partner at the company commented that these tasks were “very much in line with part of the daily activities of the consultants” involved. Notably, some forms of these tasks are also used by the company to screen job applicants, typically from elite academic backgrounds (including PhDs), for their highly coveted positions. Online Appendix G uses a survey of 11 managing directors and partners at BCG, among the most senior executives at the firm, to confirm that the underlying competencies assessed by our experimental tasks are critical for recruiting, job performance, and career progression at BCG.

Our experimental findings demonstrate mixed results regarding the quality and performance effects of humans integrating AI into their task flow. For inside-the-frontier tasks, we show that this generation of LLMs is highly capable of significantly increasing quality and productivity, leading to substantial productivity benefits across multiple dimensions when used by humans. However, quality drops substantially when knowledge workers deploy AI to solve tasks that are beyond AI’s capability frontier. Moreover, the location of the frontier of AI capability is not immediately obvious to knowledge workers, even though the tasks themselves are quite familiar. Consultants tended to overrely on AI, leading to a decline in combined human-machine performance when closer human supervision was necessary (Athey et al. 2020, Dell’Acqua 2022, Glaeser et al. 2024). This is especially relevant when we consider that our participants were highly specialized knowledge workers working within their domain of expertise, emphasizing the need for knowledge workers to have a nuanced understanding of AI’s capabilities as they navigate the evolving landscape of human-AI integration to optimize the balance between AI assistance and human expertise.

Because this jagged frontier is expanding and changing, the overall results suggest that AI will have a profound impact on knowledge work, one that will become more pronounced as LLM capabilities expand. However, the distribution of these impacts will be uneven. Our results not only highlight AI’s potential to transform knowledge work but also, underscore the importance of human involvement in machine-assisted knowledge workflows. The effectiveness of AI in knowledge work will critically depend on human judgment—particularly to discern which tasks within the workflow are suited to leveraging AI augmentation and where human expertise should be prioritized.

2. Experimental Context

BCG is a premier global management consulting firm, admitting only around 1% of applicants (Vlamis et al. 2024). Once hired, consultants enter a highly structured career track in which promotions are largely determined by strict performance appraisals. Promotion decisions often hinge on consultants’ abilities to excel in diverse tasks from creative idea generation to complex business modeling, making BCG an ideal setting to investigate how AI might affect skill development and performance. BCG also primarily promotes from within.

In close collaboration with senior BCG leaders, we designed tasks that capture core facets of consulting. These task types closely capture the breadth of work that consultants commonly handle—ranging from innovative solution proposals to methodical problem-solving. We crafted each task to reflect realistic business scenarios and crucially, ensured that some tasks were well suited to AI support and others were not as described in Section 4. To validate the real-world importance of these tasks beyond our immediate author team, we surveyed 11 managing directors and partners at BCG. Their feedback affirmed that the tasks that we designed were highly relevant for day-to-day consulting work and aligned with the core competencies that BCG evaluates in both recruiting and ongoing performance reviews (see Online Appendix G).

BCG management consultants represent an ideal population for studying knowledge work, which we define as nonroutine, cognitively demanding activities centered on the creation, application, or integration of specialized intellectual expertise.³ Online Appendix F shows how management consulting exemplifies knowledge work by analyzing Occupational Information Network (O*NET) occupational data. Although BCG consultants operate at the upper end of the complexity spectrum, our comparative analysis reveals that the diverse tasks that they perform—from analytical problem-solving to creative ideation—mirror those found across other high-skill professional domains that require a minimum of an undergraduate degree for consideration to be eligible for entry into those professions.

3. Methods

We collected data from a randomized experiment to assess the causal impact of AI, specifically GPT-4 (then the most capable of the AI models available (June 2023)) on highly qualified professionals otherwise working traditionally without AI.⁴ We preregistered our study, detailing the design structure, the experimental conditions, the dependent variables, and our main analytical approaches.⁵ We aimed to determine how introducing this AI into the tasks of highly skilled knowledge workers might augment, disrupt, or influence their performance and workflow. Details about the experimental structure are reported in Online Appendix A.

The study included 758 BCG consultants (∼7% of their individual-contributor consulting workforce) performing knowledge work. This is an elite group of knowledge workers with degrees from top universities around the world, such as Harvard, Yale, and Oxford. Tables 1 and 2 display a breakdown of demographic characteristics and experimental performance metrics for subjects across conditions. Table 3 reports the most common educational majors among our participants. Each participant completed the initial survey and was randomly assigned to one of two distinct arms of the experiment, each involving a unique type of task, with no overlap between the groups.⁶

Table 1. Summary Statistics by Demographics and Experimental Conditions

Table 2. Summary Statistics—Performance

Table 3. Most Frequently Reported Majors—Bachelor’s Degree

Approximately half of the participants (385 subjects) tackled a series of tasks that prompted them to conceptualize and develop new products, focusing on aspects such as creativity, analytical skills, persuasiveness, and writing. The other half (373 subjects) engaged in business problem-solving tasks using quantitative data, customer and company interviews, and a persuasive writing component. Both sets of tasks were developed to be realistic: that is, representative of the tasks that these workers engaged in regularly. To achieve this, the tasks were designed by BCG professionals from the relevant sectors. Figure 2 provides a visual representation of our experimental design across the two arms of the experiment.

The two arms of the experiment followed a consistent structure. Initially, participants undertook a task without the aid of AI, establishing a baseline for performance and enabling within-subject analyses.⁷ Following this, participants were randomly assigned to one of three conditions to assess the influence of AI on their tasks, with these conditions kept consistent across both arms. The first group (a control condition) proceeded without AI support. The second group (“GPT only”) had the assistance of an AI tool based on GPT-4. The third group (“GPT + overview”) utilized the same AI tool but also received a supplementary prompt engineering overview, which increased their familiarity with the AI tool. These overview materials included instructional videos and documents that outlined and illustrated effective usage strategies.

Following the baseline assessment, subjects assigned to the various AI conditions had access to a custom-built company platform for every experimental task. This platform, developed using OpenAI’s application programming interface (API), facilitated an interactive experience with OpenAI’s GPT-4 that mirrored the dynamics of ChatGPT. Although the functionality resembled that of ChatGPT, the platform enabled the collection of all data logs, including participant prompts and the AI’s corresponding responses, providing comprehensive insight into the collaborative behaviors between subjects and AI. All subjects used the same version of the tool, accessing GPT-4 as available at the end of April 2023 and using default system prompts and temperature. The same model was used for both sets of tasks.

Other than thematic differences, the two sets of tasks differed in another key way. Although both were designed to be of comparable complexity and realistically represent real-world consulting responsibilities, tasks in the first experiment were designed to be within the potential technological frontier of GPT-4. In contrast, the second experiment was deliberately designed so that human interaction and guidance were necessary to perform successfully when using GPT-4. The second experiment also contained a persuasive component that fell within the frontier.⁸ We conducted pretests for both sets of tasks, iterating several times to ensure that they were appropriately situated on either side of the frontier before initiating the experiment.

Participant engagement and effort were encouraged through performance-based incentives designed to reflect BCG’s performance evaluation system.⁹ As outlined in our preregistration, all participants who demonstrated honest effort received “office contribution” recognition—a designation with direct financial implications for annual bonuses. The bottom 5% of performers forfeited this recognition, whereas the top 20% received additional acknowledgment to their supervisors noting their exceptional performance, potentially further impacting bonus calculations and advancement prospects.

4. Results

4.1. Quality and Productivity Booster—Inside the Frontier

The inside-the-frontier experiment focused on creative product innovation and development. We defined inside‐the‐frontier tasks by pretesting them to confirm that generative AI could produce useful, high-quality outputs in the BCG client-consulting context from straightforward prompts. The initial assessment task asked participants to brainstorm innovative beverage concepts. From their set of ideas, they identified the most viable option and devised a comprehensive plan for its market debut. After this task, subjects moved to the main experimental phase, and the context transitioned to the main experimental task.

In this experimental task, participants were tasked with conceptualizing a footwear idea for niche markets and delineating every step involved from prototype description to market segmentation to entering the market. An executive from a leading global footwear company verified that task design covered the entire process that their company typically undergoes from ideation to product launch.¹⁰ Participants responded to a total of 18 tasks (or as many as they could within the given time¹¹). These tasks spanned various domains, categorizable into four types: creativity (e.g., “Propose at least 10 ideas for a new shoe targeting an underserved market or sport.”), analytical thinking (e.g., “Segment the footwear industry market based on users.”), writing proficiency (e.g., “Draft a press release marketing copy for your product.”), and persuasiveness (e.g., “Pen an inspirational memo to employees detailing why your product would outshine competitors.”). This allowed us to collect comprehensive assessments of quality. All tasks and details are reported in the Online Appendix.

The primary outcome variable in the experiment is the quality of the subjects’ responses. To quantify this quality, we employed a set of human graders to evaluate each question that participants did not leave unanswered.¹² Each response was independently evaluated by three human graders. We then calculated the mean grade assigned by humans to each response. This gave us 18 dependent variables (one per question). We subsequently averaged these scores across all questions to derive a composite “quality” score for use in our main analyses. As an additional assessment, we utilized GPT-4 to independently score subjects’ responses.¹³ Similarly to the human grades, we produced a score for each of the 18 questions and then, a composite “quality (GPT)” score.

Figure 3 uses the composite human-grader score and visually represents the performance distribution across the three experimental groups, with the average score plotted on the y axis. Comparing the dashed lines and the overall distributions of the experimental conditions in Figure 3 clearly reveals the significant performance enhancements associated with the use of GPT-4, with the GPT-only and GPT + overview groups both recording superior performances to the control group.

Table 4 presents the results of the analyses using response quality as the dependent variable and highlights the performance implications of using AI.¹⁴ Columns (1), (2), and (3) in Table 4 utilize human-generated grades as the dependent variable, whereas column (4) in Table 4 uses the composite grades generated by GPT-4. Across all specifications, both treatments—GPT + overview and GPT only—demonstrate positive effects. In column (1) in Table 4, GPT + overview leads to a 1.48 increase in scores over the control mean of 4.37, representing a 33.9% increase, with GPT only producing a 1.31 (or 29.9%) increase. Notably, columns (2), (3), and (4) in Table 4 incorporate performance metrics from the assessment task, and the treatment coefficients that they report remain very consistent.¹⁵ Column (4) in Table 4 uses GPT scores as the dependent variable and shows coefficients of 1.35 above the control group for the GPT + overview treatment and 1.22 above the control group for the GPT-only treatment, corresponding to 19% and 17.1% improvements in performance, respectively.¹⁶

Table 4. Inside the Frontier—Quality

The beneficial impacts of using AI remain consistent across all our specifications.¹⁷ As a robustness check, we isolated the quality of responses within each category (creativity, analytical thinking, writing proficiency, and persuasiveness) and applied the same model detailed in column (1) in Table 4. Each of the regressions demonstrates a significant effect of introducing AI on knowledge workers’ performance, with very sizable results for all question categories. We verified that the treatment effects are not driven by superficial spelling or grammar improvements. Adding controls for the number and rate of such errors leaves all main coefficients virtually unchanged (see Online Appendix H). Finally, we tested the reliability of these findings by replicating our analyses on a new set of grades for the ideas generated in the first question. Our results are consistent as discussed in Section 5.

Another key observation from Table 4 is the differential impact of the two AI treatments. Specifically, the GPT + overview treatment consistently exhibits a more pronounced positive effect compared with the GPT-only treatment. The bottom of Table 4 displays a p-value that tests whether the effects of receiving GPT + overview were equivalent to those of being assigned to GPT only, showing this value to be below or around the conventional 5% threshold in all specifications. This underscores the importance of the added overview for enhancing the efficacy of AI assistance, with Table 4 also highlighting various other factors, including gender, native English proficiency, tenure, location, and openness to new technology, none of which changed the significance of the impact of AI on performance.¹⁸

Table 5 presents the results related to the percentage of tasks completed by subjects, this analysis’s dependent variable. Across columns (1), (2), and (3) in Table 5, the two treatment conditions (GPT + overview and GPT) only demonstrate a positive effect on task completion. On average, these coefficients indicate a 12.2% increase in completion rates. Notably, directly comparing the two AI treatments reveals that the difference in their impacts is not statistically significant. The control group completed on average 82% of their tasks; those in the GPT + overview group completed about 93%; and those in the GPT-only group completed about 91%. Column (2) in Table 5 incorporates the performance metric from the assessment, and column (3) in Table 5 extends the analysis by including the same set of controls as in Table 4. The coefficients suggest that the integration of AI tools substantially enhances the rate of task completion while also increasing quality.

Table 5. Inside the Frontier—Completion

Figure 4 presents an important trend; the most significant beneficiaries of using AI are the bottom-half-skill subjects, consistent with findings from Noy and Zhang (2023) and Choi and Schwarcz (2024).¹⁹ By segmenting subjects exposed to one of the two AI conditions into two distinct categories—top-half-skill performers (those ranking in the top 50% on the assessment task) and bottom-half-skill performers (those in the bottom 50%)—we observed performance enhancements in the experimental task for both groups when leveraging GPT-4. Note that the top-half-skill performers also received a significant boost, although not as much as the bottom-half-skill performers.

Our findings gain additional perspective through comparison with recent research in this area. Brynjolfsson et al. (2025) explore customer support roles, identifying productivity gains primarily among lower-skilled workers, with negligible effects observed among their higher-skilled counterparts. Choi and Schwarcz (2024) use law school examinations and find that students at the bottom of the performance distribution benefited enormously from AI assistance, whereas higher-skilled students saw performance declines. Conversely, Otis et al. (2024b) report mixed outcomes from the deployment of generative AI among small-scale entrepreneurs in Kenya. Although high-performing individuals reap the benefits of AI-generated advice, low-performing individuals tend to fare worse when following AI suggestions. In our study, centered on high-end knowledge work, we observe a universal benefit for workers involved in complex, high-level tasks, particularly for tasks within the AI’s capability frontier. Within this group of elite professionals, relatively lower-skilled individuals gained the most, suggesting that generative AI may serve as a significant equalizer in the realm of high-end knowledge work, narrowing the performance gap between different skill levels; however, those who were above average also benefited meaningfully through the use of the AI system. Together, these insights highlight the significance of task-specific and contextual considerations for effectively leveraging generative AI.

For tasks within the frontier of AI capacity, we prevented subjects from completing the experiment before the allotted time was over. To achieve this, the final question was especially long, asking participants to “synthesize the insights you have gained from the previous questions and create an outline for a Harvard Business Review-style article of approximately 2,500 words.” However, although participants were required to take the full time allotted to this task, we nevertheless tracked the amount of time that they took to reach this last question, having completed the first 17 questions. Table 6 uses this Timing variable as the dependent variable. The GPT + overview treatment reduces time spent on the first 17 questions by 1,129 seconds (18.8 minutes or 22.5% faster than the control), and the GPT-only treatment reduces time spent on the first 17 questions by 1,388 seconds (23.1 minutes or 27.6% faster than the control).

Table 6. Inside the Frontier—Timing

Our results reveal significant effects, underscoring the prowess of AI even on tasks traditionally executed by highly skilled knowledge workers. Not only did using AI increase the number of subtasks completed by more than 12%, it also enhanced the quality of the responses by an average of 32%. These effects support the view that for tasks that are clearly within its frontier of capabilities, even those that historically demanded intensive human interaction, AI support provides major performance benefits.

4.2. Quality Disruptor—Outside the Frontier

In refining the task within the frontier and recognizing the substantial quality and productivity gains enabled by integrating AI, we sought to develop a task that participants could not have AI easily complete by simply copying and pasting our instructions as a prompt. We designed the beyond-the-frontier task in collaboration with BCG, using as a starting point the type of business cases that BCG uses for its highly competitive job interviews. These cases are used to gauge the skill and knowledge fit of these workers in the context of tasks of the kind that they would face in their work with the company. This meant creating a task at which knowledge workers would excel but AI would struggle, at least without extensive guidance and human intervention. We settled on a task based on an existing business case that used data from a spreadsheet and a file presenting interviews with company insiders, which were adjusted, anonymized, and adapted to the goals of this experiment. To be able to solve the task correctly, participants would have to look at the quantitative data using subtle but clear insights from the interviews. Although the spreadsheet data alone were designed to seem to be comprehensive, a careful review of the interview notes revealed crucial details. When considered in totality, this information led to an incorrect conclusion regarding what would have been provided by AI when prompted with the exercise instructions, the given data, and the accompanying interviews.

In this second experiment, the primary objective was for subjects to offer actionable strategic recommendations to a hypothetical company. First, participants worked on the assessment task, which required them to analyze the company’s sales distribution channel performance. Using insights from interviews and financial data, participants were asked to provide information and informed advice to the chief executive officer (CEO). Their recommendations were to pinpoint which channel held the most potential for growth.

Upon completing their assessment task, participants moved to the main experimental task. The focus transitioned from the examination of the company’s distribution channels to brand analysis, with subjects having to analyze the company’s brand performance. Similarly to the assessment task, participants used insights from interviews and financial data to provide recommendations to the CEO. Their recommendations were designed to pinpoint which brand held the most potential for growth. Participants were also expected to suggest actions to improve the chosen brand, regardless of the exact brand that they had chosen. Details of these tasks are reported in Online Appendix B.

For this task outside the frontier, our primary evaluation metric is “correctness,” which is represented as a binary variable, where a value of one indicates that subjects provided an accurate recommendation and zero signifies otherwise.²⁰ Figure 5 visualizes the correctness percentages for the different groups, highlighting a noticeable dip in performance among the AI treatment groups when juxtaposed with the control group. Subjects in the control group were correct about this exercise about 84.5% of the time, but the AI conditions scored at 60% and 70.6% (an average decrease of 19 percentage points when combining the AI treatment conditions and comparing them with the control condition).

Table 7 captures the impact of the AI treatments on the correctness of tasks in the outside-the-frontier experiment using linear regressions with correctness as a binary dependent variable.²¹ Both AI treatments—GPT + overview and GPT only—show a significant negative impact, with the GPT + overview group recording a more pronounced decrease (24.5% versus 13.9%). Column (2) in Table 7 introduces the performance metric from the assessment, whereas column (3) in Table 7 further refines the analysis by incorporating the same set of controls as in Tables 4 and 5. When directly comparing the two treatments, the difference in their impacts is statistically significant at the 10% level across specifications.

Table 7. Outside the Frontier—Correctness

Table 8 examines the influence of the AI treatments on the time taken by participants to complete tasks in the outside-the-frontier experiment. The dependent variable here is Timing, which represents how long subjects spent on the task calculated in seconds.²² Column (2) in Table 8 further refines the analysis by incorporating the same set of controls as in Table 4. The findings for both AI treatments—GPT + overview and GPT only—document a reduction in the time spent: more than 11 minutes less for GPT + overview (a 30% decrease in time spent compared with the control mean) and more than 6 minutes less for GPT only (a decrease of 18% in time spent compared with the control mean). Comparing the two coefficients reveals that GPT + overview recorded a more substantial—and statistically significant—decrease in time spent compared with the GPT-only group.

Table 8. Outside the Frontier—Timing

Table 9 further examines the persuasiveness and internal coherence of recommendations in the outside-the-frontier experiment, independent of whether the underlying solution was correct. The dependent variable, Subjective coherence quality, captures how well arguments were structured and supported as assessed on a 1–10 scale. Two independent sets of graders evaluated each response: BCG consultants not involved in the experiment and business school students with prior grading experience. Both groups followed a rubric developed by the authors based on workplace evaluation practices (see Online Appendix E for the full rubric). Importantly, graders were not informed of the correct strategic solution; their task was to assess the clarity, persuasiveness, and coherence of reasoning. Our subjective coherence quality metric uses the average of these two grades. Column (1) in Table 9 shows that the treatment GPT + overview leads to a 1.47-point-greater score (a 25.1% increase over the control mean), with GPT only increasing the score by 1.05 points (a 17.9% increase over the control mean). Across all specifications, subjects using AI (whether GPT + overview or GPT only) consistently outperformed those not using AI in terms of subjective coherence quality, regardless of the correctness of their answer. When we control for various factors in columns (2), (3), and (4) in Table 9, the positive impact of AI remains robust. Column (3) in Table 9 repeats the specification in column (2) in Table 9 for the subset of participants with incorrect answers, and column (4) in Table 9 does the same for those with correct answers. In both instances, the effects of using AI are positive.

Table 9. Outside the Frontier—Subjective Coherence Quality

Figure 6 illustrates subjective coherence quality conditional on whether participants ultimately solved the problem correctly. Across both the “correct” and “incorrect” subgroups, AI-supported participants produced recommendations that were more coherent and persuasive, underscoring that AI improves presentation and argumentation even when the underlying analysis is flawed. This finding underscores the multifaceted ways in which AI can influence performance in the workflow of highly skilled professionals.

Table 10 summarizes our results and presents the test statistics, p-values, and standardized effect sizes comparing the treatment conditions with the control group across our key outcome measures. Depending on the outcome variables, we use different statistical tests (t-test, chi-squared test, and MWU), which consistently show statistical significance across all our main results. Notably, all of these results remain statistically significant after applying the Bonferroni correction to adjust the significance threshold for multiple comparisons. We are additionally reporting standardized effect sizes (Cohen d, Cohen h, and rank biserial) to convey the magnitude of any significant differences. The effect sizes for our main findings range from moderate to large, with most falling in the medium to large range, underscoring the substantive impact of our interventions.

Table 10. Test Statistics and Effect Sizes—Treatments vs. Control

5. Robustness Checks with Shoe Design Experts

5.1. Individual Shoe Design Ideas

To further test the reliability of the findings presented in the main paper, we replicate our analysis on a new set of grading data. Specifically, we carry out this replication using the grades that artist evaluators assigned to shoe design ideas from the first question of the field experiment.

5.1.1. Data Collection.

The data set of shoe design ideas is derived from the first question of the “inside-the-frontier” task. Participants were asked to “generate ideas for a new shoe aimed at a specific market or sport that is underserved.” To conduct this analysis, we disaggregated each list into individual shoe design ideas, anonymized participant identifications, and presented the ideas to our graders in a randomized order.²³ Graders were instructed to rate the creativity of each idea on a scale from 1 to 10, disregarding grammatical and spelling issues. The graders were not informed about the study’s purpose or that some ideas were generated with the help of AI.

Five graders evaluated ideas. They were selected from a pool of applicants through resume screening and interviews, and they were filtered based on their relevant experience. All evaluators were students majoring in product design or industrial design at top programs with demonstrated experience in footwear design. Three graders evaluated all shoe design ideas, and two of the five graders only evaluated half of the ideas. To account for this constraint, each of these two graders was assigned a mutually exclusive, randomly selected subset comprising 50% of the data set. In total, every shoe design idea received four different evaluations by four graders.

These new grading data differ from the initial data in four important ways. First, instead of distributing 18 grades across 18 different questions as in the original “inside-the-frontier” analysis, this approach leverages at least 10 independent grades on just 1 question, thereby limiting the analysis’s scope but preserving its robustness. Second, the grades are based on creativity rather than general quality, allowing us to directly test the notion that AI tools enhance human creativity. Third, the evaluators have extensive experience in footwear design rather than a general business background, suggesting that their creativity scores hold greater validity within this specific field. Fourth, this data set uses four different evaluations instead of three.

5.1.2. Results.

The replication results are shown in Table 11.²⁴ As expected, both treatments—GPT + overview and GPT only—demonstrate positive effects across all specifications. As with the original analysis, the GPT + overview treatment exhibits a more pronounced positive effect than GPT only, although this difference is not statistically significant in any of the specifications. The magnitude of the effect is lower than in the main analysis.²⁵ In column (1) in Table 11, the GPT + overview and GPT-only treatment groups had creativity scores that were 14% and 11.7% higher than the control group, respectively, as compared with 33.9% and 29.9% in the main analysis, respectively.

Table 11. Inside the Frontier—Quality (Individual Ideas)

5.2. Full Replication with Shoe Design Experts

To further validate our findings and address feedback on the subjectivity of our evaluation metrics, we expanded our robustness checks with the shoe design experts. Although the earlier analysis focused on the creativity of ideas from the first question, this expanded analysis covers all 18 questions of the “inside-the-frontier” task.

5.2.1. Methods.

We worked with three independent shoe design experts among those who evaluated ideas in Section 5.1 to evaluate responses for all 18 questions. To ensure high-quality and thorough assessments, each question for every participant was graded by two of the three experts.²⁶ The experts were compensated on an hourly basis, and just like with our original graders, they were given unlimited time to complete their evaluations, removing any time pressure that might compromise the quality of their assessments. As with the initial analysis, graders were blind to the experimental conditions.

5.2.2. Results.

The results from this expanded expert evaluation are presented in Table 12. The findings are entirely consistent with the results reported in the main paper and in fact, demonstrate even stronger positive effects of AI on performance. Across all specifications, both the GPT + overview and GPT-only treatments show a significant and substantial improvement in the quality of work produced by participants. The effect sizes are larger than those observed in the main analysis, reinforcing our conclusion that AI assistance enhances the quality of knowledge work on tasks inside the frontier. These positive effects remain statistically significant when the analysis is disaggregated by question type (creativity, analytical thinking, writing proficiency, and persuasiveness).

Table 12. Inside the Frontier—Quality (Design Experts)

These robustness checks support our main findings. Additionally, this convergence across diverse evaluator types and methodologies provides strong evidence against systematic evaluator bias and the hypothesis that graders are systematically fooled by AI-generated content.

6. Discussion

In this paper, we study the use of AI with a jagged capability frontier by humans performing high-end knowledge work. Using a preregistered, randomized laboratory-in-the-field experiment, we test the potential dual role of AI in augmenting the efforts of knowledge workers, recognizing that it functions as both a booster—enhancing efficiency and productivity on tasks within its capability frontier—and a disruptor, which negatively impacts performance on tasks beyond its frontier. Our findings underscore the transformative potential of AI and offer insights into harnessing its capabilities for optimal outcomes.

A crucial feature of our experiment was the nature of our experimental subjects. Specifically, we were able to recruit from a highly skilled and motivated population, with participants who engaged in tasks that were developed in collaboration with our partner company. These tasks closely mirrored their professional activities as knowledge workers, allowing us to assess AI’s impact on tasks that approximate real-world workflows while recognizing that they do not fully capture every complexity of on-the-job work.

We found that the utility of AI can fluctuate across a professional’s workflow, with some tasks falling inside the frontier, whereas others fall outside of it. Figure 7 displays the treatment effect sizes in percentage change for both inside and outside the frontier, showing the magnitude of our treatment effects conditions across all of our key outcome measures. For tasks inside the frontier, our findings have substantial and positive performance implications. Across 18 realistic business tasks—ranging from creative to analytical tasks—AI significantly improved performance and quality for every model specification, increasing speed by more than 25%, performance by more than 30%, and task completion by more than 12%. However, for a task outside the frontier, subjects using AI were 19 percentage points less likely to produce correct solutions. Although this design highlights a single outside-the-frontier task—an inherent limitation—it underscores that even a single complex challenge beyond AI’s current capabilities can disproportionately reduce overall performance if users rely too heavily on AI’s output. These results suggest a need to understand the frontier’s shape and position as well as how it is perceived by knowledge workers to better determine AI’s influence on work.

Our design employs different evaluation methods by task type, directly mirroring how consulting work is assessed in practice. For writing-intensive activities inside the frontier, performance is evaluated through subjective rubrics, matching BCG’s standard procedures because no single correct answer exists. Conversely, the strategy case outside the frontier features an objectively correct recommendation that is measured dichotomously and supplemented with separate persuasiveness assessments. This hybrid approach reflects again the incommensurability of knowledge work outputs. Nevertheless, subjective metrics may contain evaluation bias, representing a key limitation of our study and suggesting important avenues for developing additional objective performance proxies in future research.

Our findings should be interpreted within important boundary conditions regarding incentive structures. Our results apply most directly to knowledge work contexts where quality within allocated time frames is prioritized over speed optimization, such as is the case at BCG and similar professional service environments. In settings where efficiency gains translate directly into rewards or where explicit accuracy-speed trade-offs exist—such as manufacturing, customer service with call-time metrics, or piecework environments—the effects of AI deployment may differ substantially. Future research should investigate how the jagged frontier manifests across different incentive structures in knowledge work.

A further caveat concerns who actually chooses to adopt generative AI when its use is discretionary rather than strongly encouraged, such as in our setting. Recent evidence shows sizable heterogeneity; early adopters are disproportionately male and more technologically confident, and the resulting performance gains can widen pre-existing gender- and skill-gaps rather than close them (Otis et al. 2024a, Humlum and Vestergaard 2025, Blandin et al. 2026). If such adoption gaps persist, the technology could widen existing promotion differentials. Additionally, beyond unequal gains, AI may substitute some workers entirely. Recent evidence from online labor markets indicates that generative AI has reduced both employment and earnings for freelancers in affected occupations (Hui et al. 2024), underscoring that AI’s impact on work encompasses not only how effectively workers perform tasks but also, their continued employment prospects.

Although our analysis focuses on individuals, the scale of our experiment within a single organization surfaces implications for how firms might integrate AI into knowledge work. Our findings suggest that organizations should focus on the knowledge workflow and the tasks within it, and for each task, they should evaluate the value of using different configurations of humans and AI. This will require rethinking collaboration between humans and AI (Raisch and Krakowski 2021, Faraj and Leonardi 2022, Feuerriegel et al. 2022, Lebovitz et al. 2022, Anthony et al. 2023, Choudhary et al. 2023, Dell’Acqua et al. 2025), human training (Beane 2019, Cowgill et al. 2020, Kellogg et al. 2021, Gaessler and Piezunka 2023), how new roles will emerge and be created (Barrett et al. 2012, Sergeeva et al. 2020, Allen and Choudhury 2022, Kellogg 2022), new capabilities and strategies (Iansiti and Lakhani 2020), and new forms of organizing (Bailey et al. 2022, Beane and Leonardi 2025). In this context, assessing AI capabilities relative to tasks within workflows becomes especially important (Lebovitz et al. 2021, Eloundou et al. 2024).

This paper contributes to the emerging literature on the impact of AI on work by showing its nuanced impact on knowledge work. Studies such as Noy and Zhang (2023) have documented that providing AI assistance boosts productivity and efficiency among college-educated professionals. Our paper adds substantially to this finding in several key respects. First, our laboratory-in-the-field experiment conducted with management consultants offers a real-world setting that mirrors the complexities of actual professional workflows, lending high ecological validity to our results. Second, whereas Noy and Zhang (2023) focused primarily on writing tasks, we include a broader range of activities. Third, we designed a dual-task framework with tasks outside current AI capabilities. This “jagged frontier” approach reveals that although AI significantly improves performance on tasks within its capabilities, it can hinder performance on those requiring complex analysis or reasoning.

Other papers have illustrated AI’s heterogeneous effects. For example, Brynjolfsson et al. (2025) have highlighted AI’s varying impact based on skill levels, showing enhancements primarily among lower-skilled workers, whereas Otis et al. (2024b) have found no overall effect from AI usage but noted benefits disproportionately affecting higher-skilled individuals. Our study, the first to measure the impact of AI on knowledge workers in a setting resembling their standard work activities, addresses the tension of the contrasting results in the literature by demonstrating AI’s both positive and negative effects on knowledge work. These enhancements show that our work not only confirms the productivity benefits of introducing AI but also, expands our understanding of AI’s limitations and the best ways to integrate it in professional environments.

Our findings also connect to several related literatures. Prior work shows that AI systems tend to generate larger performance gains in ideation and content creation tasks than in judgment or decision-making tasks (e.g., Vaccaro et al. 2024). Although our findings are consistent with this pattern, they show that such divergent effects can arise within the same knowledge workflow and for the same professionals, complicating knowledge workers’ and managers’ ability to anticipate when AI assistance will be beneficial versus harmful. Our findings also relate to research on automation bias, which documents how human performance can deteriorate when automated systems provide imperfect guidance in decision-making contexts (e.g., Skitka et al. 1999, Buçinca et al. 2021). We highlight how such performance losses can emerge when AI systems perform exceptionally well on adjacent tasks, underscoring the challenge posed by AI’s jagged capabilities.

Our findings offer multiple avenues for interpretation in the context of future implementations of human-AI collaboration. First, our results lend support to optimism about AI capabilities for important high-end knowledge work tasks. In our study, the human-AI combination proved surprisingly capable, demonstrating the potential of AI-aided professional work. However, we also show the challenges resulting from AI’s jagged capabilities; experienced and incentivized knowledge professionals, engaged in tasks akin to some of their daily responsibilities, performed worse when given access to AI. This happened even though they did not need to rely on AI’s output to complete the task. The jagged nature of the knowledge frontier suggests that in real-world settings, knowledge-intensive workflows will most likely straddle both sides, creating situations where knowledge workers—unaware of their position relative to AI’s capabilities—perform worse as a result of becoming overly reliant on AI (“falling asleep at the wheel” (Dell’Acqua 2022)), a risk that is reinforced by the persuasive capabilities of generative AI (Randazzo et al. 2025a).

Since the release of our working paper in September 2023, the notion of AI as a jagged technology has been acknowledged by industry leaders and practitioners. Sundar Pichai, the CEO of Google, noted in an interview with Lex Fridman (Pichai 2025): “Have you heard AJI, the artificial jagged intelligence? Sometimes feels that way, both their progress and you see what they can do and then you can trivially find they make numerical errors or counting R’s in strawberry or something, which seems to trip up most models or whatever it is. So maybe we should throw that term in there. I feel like we are in the AJI phase where dramatic progress, some things don’t work well, but overall you’re seeing lots of progress.” Similarly, Andrej Karpathy, one of the leading AI systems developers, emphasized the practical implications of this phenomenon, advising practitioners to “use LLMs for the tasks they are good at but be on a lookout for jagged edges, and keep a human in the loop” (Karpathy 2024). In a subsequent blog post, Karpathy (2025) elaborated on this concept: “As verifiable domains allow for [reinforcement learning from verifiable rewards], LLMs ‘spike’ in capability in the vicinity of these domains and overall display amusingly jagged performance characteristics—they are at the same time a genius polymath and a confused and cognitively challenged grade schooler, seconds away from getting tricked by a jailbreak to exfiltrate your data.”

As the boundaries of AI capabilities continue to expand, human professionals may need to recalibrate their understanding of the frontier. The frontier is jagged, but it is not fixed, and any given task may move inside with the release of a new model. Overall, AI appears poised to significantly affect human cognition and problem-solving but in uneven ways. AI may lower—and raise—the cognitive costs associated with knowledge acquisition and application across tasks. Effectively leveraging AI in professional settings will require understanding how work is positioned relative to AI’s jagged frontier, even while subject to quadratic and exponential increases in benchmark performance on a periodic basis.

Author Contributions

FDA and KRL established the collaboration, designed the experiment, and oversaw its execution. FDA and KRL led the analyses, framing, and write-up. FC was instrumental in initiating the collaboration. EMcF contributed to variable construction and all analyses. EM contributed to the design, the prompt training and the framing. HL and KK led the qualitative component of the study and contributed to the framing of the paper. FC, LK and SR facilitated organizational access, enabled the execution of the experiment, and supported selected analyses. All coauthors contributed to the experimental design and to revisions of the manuscript.