The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers

References

• Acemoglu D (2025) The simple macroeconomics of AI. Econom. Policy 40(121):13–58.Crossref, Google Scholar
• Agrawal A, Gans J, Goldfarb A (2019) Economic policy for artificial intelligence. Innovation Policy Econom. 19(1):139–159.Crossref, Google Scholar
• Angelopoulos P, Lee K, Misra S (2024) Causal alignment: Augmenting language models with A/B tests. Preprint, submitted April 15, https://doi.org/10.2139/ssrn.4781850.Google Scholar
• Baily MN, Brynjolfsson E, Korinek A (2023) Machines of Mind: The Case for an AI-Powered Productivity Boom (Brookings Institute, Washington, DC).Google Scholar
• Bick A, Blandin A, Deming DJ (2024) The rapid adoption of generative AI. NBER Working Paper No. 32966, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Bloom N, Eifert B, Mahajan A, McKenzie D, Roberts J (2012) Does management matter? Evidence from India. Quart. J. Econom. 128(1):1–51.Crossref, Google Scholar
• Brand J, Israeli A, Ngwe D (2023) Using GPT for market research. Working Paper No. 23-062, Harvard Business School Marketing Unit, Cambridge, MA.Google Scholar
• Bresnahan T (2024) What innovation paths for AI to become a GPT? J. Econom. Management Strategy 33(2):305–316.Crossref, Google Scholar
• Brynjolfsson E, Li D, Raymond L (2025) Generative AI at work. Quart. J. Econom. 140(2):889–942.Crossref, Google Scholar
• Brynjolfsson E, Rock D, Syverson C (2021) The productivity J-curve: How intangibles complement general purpose technologies. Amer. Econom. J. Macroeconom. 13(1):333–372.Crossref, Google Scholar
• Bubeck S, Chandrasekaran V, Eldan R, Gehrke J, Horvitz E, Kamar E, Lee P, et al. (2023) Sparks of artificial general intelligence: Early experiments with GPT-4. Preprint, submitted March 22, https://arxiv.org/abs/2303.12712.Google Scholar
• Campero A, Vaccaro M, Song J, Wen H, Almaatouq A, Malone TW (2022) A test for evaluating performance in human-computer systems. Preprint, submitted June 24, https://arxiv.org/abs/2206.12390.Google Scholar
• Coussens S, Spiess J (2021) Improving inference from simple instruments through compliance estimation. Preprint, submitted August 8, https://arxiv.org/abs/2108.03726.Google Scholar
• Cowgill B, Dell’Acqua F, Deng S, Hsu D, Verma N, Chaintreau A (2020) Biased programmers? Or biased data? A field experiment in operationalizing AI ethics. Proc. 21st ACM Conf. Econom. Comput. (Association for Computing Machinery, New York), 679–681.Google Scholar
• Dell’Acqua F, McFowland E, Mollick ER, Lifshitz-Assaf H, Kellogg K, Rajendran S, Krayer L, et al. (2023) Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Working Paper No. 24-013, Harvard Business School Technology & Operations Management Unit, Cambridge, MA.Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2015) Algorithm aversion: People erroneously avoid algorithms after seeing them err. J. Experiment. Psych. General 144(1):114.Crossref, Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2018) Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Sci. 64(3):1155–1170.Link, Google Scholar
• Eloundou T, Manning S, Mishkin P, Rock D (2024) GPTs are GPTs: Labor market impact potential of LLMs. Science 384(6702):1306–1308.Crossref, Google Scholar
• Emanuel N, Harrington E, Pallais A (2023) The power of proximity to coworkers: Training for tomorrow or productivity today? NBER Working Paper No. 31880, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Frank MR, Autor D, Bessen JE, Brynjolfsson E, Cebrian M, Deming DJ, Feldman M, et al. (2019) Toward understanding the impact of artificial intelligence on labor. Proc. Natl. Acad. Sci. USA 116(14):6531–6539.Crossref, Google Scholar
• Furman J, Seamans R (2019) AI and the economy. Innovation Policy Econom. 19(1):161–191.Crossref, Google Scholar
• Giles DEA (1984) Instrumental variables regressions involving seasonal data. Econom. Lett. 14(4):339–343.Crossref, Google Scholar
• Goli A, Singh A (2024) Frontiers: Can large language models capture human preferences? Marketing Sci. 43(4):709–722.Link, Google Scholar
• Greenstein S, Lovin N, Wallsten S, Herman K, Pinckney S (2024) A guide to the vocabulary evolution and impact of artificial intelligence (AI). Working Paper No. 625-039, Harvard Business School, Cambridge, MA.Google Scholar
• Gui G, Toubia O (2023) The challenge of using LLMs to simulate human behavior: A causal inference perspective. Preprint, submitted December 24, https://arxiv.org/abs/2312.15524.Google Scholar
• Hoffmann M, Boysel S, Nagle F, Peng S, Xu K (2024) Generative AI and the nature of work. Working Paper No. 11479, Harvard Business School Publishing, Boston, MA.Google Scholar
• Huntington-Klein N (2020) Instruments with heterogeneous effects: Bias, monotonicity, and localness. J. Causal Inference 8(1):182–208.Crossref, Google Scholar
• Jaffe S, Shah NP, Butler J, Farach A, Cambon A, Hecht B, Schwarz M, et al. (2024) Generative AI in Real-World Workplaces (Microsoft Redmond, WA), 1–10.Google Scholar
• Meyer J (2011) Workforce age and technology adoption in small and medium-sized service firms. Small Bus. Econom. 37(3):305–324.Crossref, Google Scholar
• Microsoft Corporation (2025) Microsoft fiscal year 2024 Q2 earnings call. Accessed January 20, 2026, https://www.microsoft.com/en-us/investor/events/fy-2024/earnings-fy-2024-q2.Google Scholar
• Murciano-Goroff R (2022) Missing Women in tech: The labor market for highly skilled software engineers. Management Sci. 68(5):3262–3281.Link, Google Scholar
• Nagle F, Greenstein S, Roche M, Wright NL, Mehta S (2023) CoPilots(s): Generative AI at Microsoft and GitHub. Harvard Bus. School Case 9:624–010.Google Scholar
• Noy S, Zhang W (2023) Experimental evidence on the productivity effects of generative artificial intelligence. Science 381(6654):187–192.Crossref, Google Scholar
• Otis N, Clarke R, Delecourt S, Holtz D, Koning R (2024) The uneven impact of generative AI on entrepreneurial performance. Working paper, No. 24-042, Harvard Business School, Cambridge, MA.Google Scholar
• Peng S (2024) The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers. Registered randomized controlled trial (RCT) entry, AEA RCT Registry (AEARCTR-0014530). Initial registration date: October 7, 2024.Google Scholar
• Peng S, Kalliamvakou E, Cihon P, Demirer M (2023) The impact of AI on developer productivity: Evidence from GitHub Copilot. Preprint, submitted February 13, https://arxiv.org/abs/2302.06590.Google Scholar
• Vaithilingam P, Zhang T, Glassman EL (2022) Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. Proc. CHI Conf. Human Factors Comput. Systems Extended Abstracts (Association for Computing Machinery, New York), 1–7.Google Scholar
• Ye Z, Yoganarasimhan H, Zheng Y (2025) LOLA: LLM-assisted online learning algorithm for content experiments. Marketing Sci. 44(5):995–1016.Abstract, Google Scholar
• Yeverechyahu D, Mayya R, Oestreicher-Singer G (2024) The impact of large language models on open-source innovation: Evidence from GitHub Copilot. Preprint, submitted September 12, https://arxiv.org/abs/2409.08379.Google Scholar