Align Generative Artificial Intelligence with Human Preferences: A Novel Large Language Model Fine-Tuning Method for Online Review Management

Online reviews have played a pivotal role in consumers’ decision-making processes. Existing research has highlighted the significant impact of managerial review responses on customer relationship management and firm performance. However, large portions of online reviews remain unaddressed because of the considerable human labor required to respond to the rapid growth of online reviews. Although generative artificial intelligence (AI; especially large language models (LLMs)) has achieved remarkable success in a range of tasks, they (i.e., generative AI, such as GPT-4) are general-purpose models and may not align well with domain-specific human preferences. To tailor these general generative AI models to domain-specific applications, fine-tuning is commonly employed. Nevertheless, several challenges persist in fine-tuning with domain-specific data, including hallucinations, difficulty in representing domain-specific human preferences, and overconservatism in offline policy optimization. To address these challenges, we propose a novel preference fine-tuning method to align an LLM with domain-specific human preferences for generating online review responses. Specifically, we first identify the source of hallucination and propose an effective context-augmentation approach to mitigate the LLM hallucination. To represent human preferences, we propose a novel theory-driven preference fine-tuning approach that automatically constructs human preference pairs in the online review domain. Additionally, we propose a curriculum learning approach to further enhance preference fine-tuning. To overcome the challenge of overconservatism in the existing offline preference fine-tuning method, we propose a novel density estimation-based support-constraint method to relax the conservatism, and we mathematically prove its superior theoretical guarantees. Extensive evaluations employing objective evaluation metrics, human assessments, and qualitative analyses substantiate the superiority of our proposed preference fine-tuning method. For practical deployment in real-world systems, we recommend two types of hybrid approaches to synergize human and LLM capabilities, which can significantly reduce human labor and time in responding to online reviews.

History: Olivia Liu Sheng, Senior Editor; Huimin Zhao, Associate Editor.

Funding: This work was supported by the Division of Information and Intelligent Systems and the National Science Foundation [Grant 1844983].

Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2024.1518.