Frontiers: Reducing Disparity Between LLMs and Humans: Optimal LLM Sample Calibration

Large language models (LLMs) are increasingly used to measure preferences and opinions, but their responses can diverge from those of the human populations they are meant to represent. Analysts typically rely on a single aggregate prompt or combine multiple LLM outputs using heuristic rules, such as uniform averaging, population-weighted poststratification, or selecting the best single subgroup on a calibration split. These approaches can remain misaligned with human benchmarks. We propose human mimicry calibration (HuMCal), a post hoc, model-agnostic method that learns how to combine LLM outputs from different prompts, personas, or model checkpoints. Our method uses aggregate human benchmark data and requires neither access to model internals nor separate human benchmarks for each subgroup. On the calibration data, HuMCal chooses the weighted average that best matches the human benchmark. When solved exactly, its calibration loss is weakly lower than that of any fixed-weight rule over the same set of outputs, including uniform weighting, poststratification, and the best single subgroup. We evaluate HuMCal in a revealed-preference market-structure application using U.S. bank card transactions and in a public opinion application using Pew survey data. Across both settings, HuMCal improves agreement on the calibration split relative to all evaluated fixed-weight baselines. Out of sample, the learned weights outperform heuristic baselines in most evaluated transfers across time, geography, industry, and question clusters. In the market-structure application, recalibration with newer LLM checkpoints improves agreement in all evaluated settings. These results suggest that HuMCal can serve as a scalable calibration tool for LLM-based measurement when collecting fresh benchmark data for each target setting is costly.

History: Olivier Toubia served as the senior editor.

Funding: This work was supported by a McCombs Research Excellence Grant and the University of Texas at Austin Office of the Vice President for Research & Creative Grants. Y. Leng is supported by the U.S. National Science Foundation Division of Information and Intelligent Systems [Grant IIS-2153468].

Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mksc.2024.0983.