Frontiers in Operations: A Moment for Reflection: Debiasing Service Evaluations

Problem definition: Service quality is assessed with objective and subjective measures. Objective measures include queue length/wait times, service times, and service failures, whereas subjective measures include service evaluations and social media commentary. Subjective measures are often easier to collect, requiring no sophisticated tracking technology, and can capture multiple dimensions of the service experience in a single response. However, subjective data are prone to bias. We focus on bridging the gap between objective and subjective measures of service quality by reducing bias in service evaluations. The structure of service evaluations is consistent across organizations: star ratings are collected before comments. We propose a simple intervention—reversing this order—to mitigate bias in star ratings because the process of writing provides time and space for the evaluator to reflect on their experience. Methodology/results: We conducted an experiment where participants received a sequence-based service, with the same overall level of service, from servers who varied in their demographic characteristics. Following the service, participants evaluated the performance of the servers with the order of star ratings and comments randomized. We find no evidence of demographic bias toward the servers but find that the sequence of good and bad experiences in the service leads to biased star ratings. Most importantly, we find that collecting comments prior to star ratings mitigates the sequential bias in star ratings because of participants reflecting on the service experience. Managerial implications: We show that star ratings can be biased if collected in the traditional manner and that this bias can be reduced if comments are collected first. Implementing this change to service evaluations could help to ensure that the job performance of human servers is more accurately and fairly assessed. For artificial intelligence service systems (i.e., AI servers), this change can provide a simple way to debias training data.

History: This paper has been accepted in the Manufacturing & Service Operations Management Frontiers in Operations Initiative.

Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2025.0060.