The Role of Expectations and Explicit Accuracy Information in the Utilization of Advice From Human-in-the-loop Systems

Human-in-the-loop (HiL) systems integrate human judgment with algorithmic output, combining computational processing with human oversight and expertise. Despite their growing prevalence as sources of advice in organizational decision making, the extent to which decision makers value HiL advice remains underexplored, particularly compared to the extensive literature on purely algorithmic advice. Using the Judge Advisor System paradigm, we compare advice from three sources, HiL systems, algorithms, and human experts, and systematically manipulate the provision of explicit accuracy information. Our findings are threefold. First, participants hold higher expectations of accuracy for HiL systems relative to algorithms and human experts, yet this does not lead to higher utilization of HiL advice. Second, explicit accuracy information increases advice utilization for all sources: more accurate sources are used more frequently, regardless of type. Third, within any given source, higher perceived accuracy correlates with greater advice utilization. These results suggest a clear path forward for organizations: HiL systems will be valued when they deliver on their promise of superior accuracy by leveraging complementary human and algorithmic capabilities. Organizations should focus on designing HiL systems that combine the unique strengths of each source, such as algorithmic data processing with human judgment in ambiguous environments, and on transparently communicating accuracy information. Successful adoption of HiL systems requires empirical evidence of accuracy gains in specific contexts, not assumptions about decision maker preferences for HiL systems.