Unraveling Generative AI from a Human Intelligence Perspective: A Battery of Experiments

Generative artificial intelligence and large language models (LLMs) are advancing rapidly, yet assessing their intelligence from a human-centric viewpoint remains underexplored. This perspective is crucial as intelligence is deeply rooted in human society, fundamentally shaping a wide array of job roles, creative activities, and human interactions and organization forms. We propose a new framework for comprehensively understanding LLMs holistic intelligence based on behavioral theory and experiments, using human intelligence as a benchmark. Using our framework, a comparison between a diverse group of human participants and LLMs through extensive online experiments reveals that GPT-4 surpasses human levels in cognitive, creative, and emotional aspects but falls short in social intelligence, particularly in understanding social interest, demonstrating self-efficacy, and interpreting mental states. Furthermore, we present several supporting contexts to demonstrate and validate the practical value of our framework. We validate our intelligence-based framework by assessing GPT-4’s impact on jobs across various zones and benchmarking it against an established study. Results show that our framework not only produces consistent findings but uniquely explain the underlying rationale behind LLM job impacts, affirming its validity and practical relevance. Moreover, we showcase the framework’s broader applicability as a reusable measurement system for firms and policymakers to understand LLM intelligence and forecast its impact on jobs for any LLM. Through multiple practical scenarios illustrations, we demonstrate how firms and policymakers can leverage this tool to make informed decisions on where to and where not to adopt LLMs, selecting integration strategies for different job types, and identifying the most suitable model by comparing multiple LLMs to meet specific job and business requirements. Overall, our study provides a valuable tool to facilitate the effective integration of LLMs into human society, supporting strategic decision making and policymaking.

History: Sean Xu, Senior Editor; Kevin Hong, Associate Editor.

Funding: T. Sun acknowledges research support from CKGSB Research Institute, Center for Digital Transformation and ABRI Grant.

Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2023.0487.