Help
Cart
Skip main navigation

Available Issues

April 12, 2013 - June 15, 2026

Available Issues

April 12, 2013 - June 15, 2026

Human-Algorithm Collaborative Truth Inference in Crowdsourcing

Published Online:15 Sep 2025https://doi.org/10.1287/ijoc.2023.0440

References

  • Amershi S, Weld D, Vorvoreanu M, Fourney A, Nushi B, Collisson P, Suh J, et al. (2019) Guidelines for human-AI interaction. Brewster S, Fitzpatrick G, Cox A, Kostakos V, eds. Proc. 2019 CHI Conf. Human Factors Comput. Systems (Association for Computing Machinery, New York), 1–13.Google Scholar
  • Bi W, Wang L, Kwok JT, Tu Z (2014) Learning to predict from crowdsourced data. Zhang N, Tian J, eds. Proc. 30th Conf. Uncertainty Artificial Intelligence (Association for Uncertainty in Artificial Intelligence, Arlington, VA), 82–91.Google Scholar
  • Blei DM, Kucukelbir A, McAuliffe JD (2017) Variational inference: A review for statisticians. J. Amer. Statist. Assoc. 112(518):859–877.Crossref, Google Scholar
  • Cheng J, Bernstein MS (2015) Flock: Hybrid crowd-machine learning classifiers. Cosley D, Forte A, Ciolfi L, McDonald D, eds. Proc. 18th ACM Conf. Comput. Supported Cooperative Work Soc. Comput. (Association for Computing Machinery, New York), 600–611.Google Scholar
  • Davis J, Goadrich M (2006) The relationship between precision-recall and ROC curves. Cohen W, Moore A, eds. Proc. 23rd Internat. Conf. Machine Learn. (Association for Computing Machinery, New York), 233–240.Google Scholar
  • Dawid AP, Skene AM (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm. J. Royal Statist. Soc. Ser. C (Appl. Statist.) 28(1):20–28.Google Scholar
  • Ding X, Liu T, Duan J, Nie JY (2015) Mining user consumption intention from social media using domain adaptive convolutional neural network. Proc. AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 2389–2395.Google Scholar
  • Dizaji KG, Huang H (2018) Sentiment analysis via deep hybrid textual-crowd learning model. McIlraith S, Weinberger K, eds. Proc. AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 1563–1564.Google Scholar
  • Fang Y, Wei X, Chen W (2025) WILC: A wisdom integration framework for LLM crowds. Myers M, Alias R, Boh W, Davison R, Tan B, Rahim N, eds. PACIS 2025 Proc. (Association for Information Systems, Atlanta).Google Scholar
  • Fügener A, Grahl J, Gupta A, Ketter W (2021) Will humans-in-the-loop become borgs? Merits and pitfalls of working with AI. MIS Quart. 45(3):1527–1556.Crossref, Google Scholar
  • Ji Q, Jiang L, Zhang W (2023) Dual-view noise correction for crowdsourcing. IEEE Internet Things J. 10(13):11804–11812.Crossref, Google Scholar
  • Johnson MJ, Duvenaud DK, Wiltschko A, Adams RP, Datta SR (2016) Composing graphical models with neural networks for structured representations and fast inference. Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, eds. Proc. 30th Internat. Conf. Neural Inform. Processing Systems (Curran Associates, Red Hook, NY), 2954–2962.Google Scholar
  • Kamar E, Hacker S, Horvitz E (2012) Combining human and machine intelligence in large-scale crowdsourcing. Hoek W, Padgham L, Conitzer V, Winikoff M, eds. Proc. 11th Internat. Conf. Autonomous Agents Multiagent Systems (International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC), 467–474.Google Scholar
  • Kazai G, Kamps J, Milic-Frayling N (2011) Worker types and personality traits in crowdsourcing relevance labels. Berendt B, de Vries A, Fan W, Macdonald C, Ounis I, Ruthven I, eds. Proc. 20th ACM Internat. Conf. Inform. Knowledge Management (Association for Computing Machinery, New York), 1941–1944.Google Scholar
  • Kim HC, Ghahramani Z (2012) Bayesian classifier combination. Lawrence N, Girolami M, eds. Proc. 15th Internat. Conf. Artificial Intelligence Statist. (Proceedings of Machine Learning Research, New York), 619–627.Google Scholar
  • Kingma DP, Welling M (2014) Auto-encoding variational Bayes. 2nd Internat. Conf. Learn. Representations, 1–14.Google Scholar
  • Li H, Yu B (2014) Error rate bounds and iterative weighted majority voting for crowdsourcing. Preprint, submitted November 15, https://arxiv.org/abs/1411.4086.Google Scholar
  • Li J, Sun H, Li J (2023) Beyond confusion matrix: Learning from multiple annotators with awareness of instance features. Machine Learn. 112(3):1053–1075.Crossref, Google Scholar
  • Li SY, Jiang Y, Chawla NV, Zhou ZH (2018) Multi-label learning from crowds. IEEE Trans. Knowledge Data Engrg. 31(7):1369–1382.Crossref, Google Scholar
  • Li C, Zhang Z, Saugstad M, Safranchik E, Kulkarni C, Huang X, Patel S, Iyer V, Althoff T, Froehlich JE (2024) LabelAId: Just-in-time AI interventions for improving human labeling quality and domain knowledge in crowdsourcing systems. Mueller FF, Kyburz P, Williamson JR, Sas C, Wilson ML, Dugas PT, Shklovski I, eds. Proc. 2024 CHI Conf. Human Factors Comput. Systems (Association for Computing Machinery, New York), 1–21.Google Scholar
  • Luo Y, Tian T, Shi J, Zhu J, Zhang B (2018) Semi-crowdsourced clustering with deep generative models. Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, eds. Proc. 32nd Internat. Conf. Neural Inform. Processing Systems (Curran Associates, Red Hook, NY), 3216–3226.Google Scholar
  • Matsubara M, Borromeo RM, Amer-Yahia S, Morishima A (2021) Task assignment strategies for crowd worker ability improvement. Proc. ACM Human-Comput. Interaction 5(CSCW2):1–20.Crossref, Google Scholar
  • Mitchell TM (1980) The need for biases in learning generalizations. Rutgers CS Tech Report CBM-TR-117, Rutgers University, New Brunswick, NJ.Google Scholar
  • Moreno PG, Artés-Rodríguez A, Teh YW, Perez-Cruz F (2015) Bayesian nonparametric crowdsourcing. J. Machine Learn. Res. 16(1):1607–1627.Google Scholar
  • Rodrigues F, Pereira F (2018) Deep learning from crowds. McIlraith S, Weinberger K, eds. Proc. AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 1611–1618.Crossref, Google Scholar
  • Rodrigues F, Lourenco M, Ribeiro B, Pereira FC (2017) Learning supervised topic models for classification and regression from crowds. IEEE Trans. Pattern Anal. Machine Intelligence 39(12):2409–2422.Crossref, Google Scholar
  • Russakovsky O, Li LJ, Fei-Fei L (2015) Best of both worlds: Human-machine collaboration for object annotation. Proc. IEEE Conf. Comput. Vision Pattern Recognition (IEEE Computer Society, New York), 2121–2131.Google Scholar
  • Sheng VS, Zhang J (2019) Machine learning with crowdsourcing: A brief summary of the past research and future directions. Proc. AAAI Conf. Artificial Intelligence (AAAI Press, Washington, DC), 9837–9843.Crossref, Google Scholar
  • Simpson ED, Venanzi M, Reece S, Kohli P, Guiver J, Roberts SJ, Jennings NR (2015) Language understanding in the wild: Combining crowdsourcing and machine learning. Gangemi A, Leonardi S, Panconesi A, eds. Proc. 24th Internat. Conf. World Wide Web (International World Wide Web Conferences Steering Committee, Geneva), 992–1002.Google Scholar
  • Van Engelen JE, Hoos HH (2020) A survey on semi-supervised learning. Machine Learn. 109(2):373–440.Crossref, Google Scholar
  • Wainwright MJ, Jordan MI (2008) Graphical models, exponential families, and variational inference. Foundations Trends Machine Learn. 1(1–2):1–305.Google Scholar
  • Wang J, Ipeirotis PG, Provost F (2017) Cost-effective quality assurance in crowd labeling. Inform. Systems Res. 28(1):137–158.Link, Google Scholar
  • Wei X (2020) Data science-driven crowd intelligence and its business applications. Ph.D. thesis, The University of Arizona, Tucson.Google Scholar
  • Wei X, Zhang Z, Zhang M, Chen W, Zeng DD (2022) Combining crowd and machine intelligence to detect false news on social media. MIS Quart. 46(2):977–1008.Crossref, Google Scholar
  • Wei X, Zhang M, Zhang Q, Li Z, Zeng DD (2025) Human-algorithm collaborative truth inference in crowdsourcing. https://doi.org/10.1287/ijoc.2023.0440.cd, https://github.com/INFORMSJoC/2023.0440.Google Scholar
  • Welinder P, Branson S, Mita T, Wah C, Schroff F, Belongie S, Perona P (2010) Caltech-UCSD Birds 200 (California Institute of Technology, Pasadena).Google Scholar
  • Yin J, Luo J, Brown SA (2021) Learning from crowdsourced multi-labeling: A variational Bayesian approach. Inform. Systems Res. 32(3):752–773.Abstract, Google Scholar
  • Yin L, Liu Y, Zhang W, Yu Y (2020) Truth inference with a deep clustering-based aggregation model. IEEE Access 8:16662–16675.Crossref, Google Scholar
  • Zhang J (2022) Knowledge learning with crowdsourcing: A brief review and systematic perspective. IEEE/CAA J. Automatica Sinica 9(5):749–762.Crossref, Google Scholar
  • Zhang J, Wu X, Sheng VS (2016a) Learning from crowdsourced labeled data: A survey. Artificial Intelligence Rev. 46(4):543–576.Crossref, Google Scholar
  • Zhang M, Xu Y, Wei X (2024) AI-centered vs. human-centered: Exploring users’ attitude toward AIGC in varying forms of human-AI collaboration. Phan T, Tan B, Su L, Thuan N, Chau M, Goh K, eds. PACIS 2024 Proc. (Association for Information Systems, Atlanta).Google Scholar
  • Zhang Y, Chen X, Zhou D, Jordan MI (2016b) Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. J. Machine Learn. Res. 17(1):3537–3580.Google Scholar
  • Zhang Z, Wei X, Zheng X, Li Q, Zeng DD (2022) Detecting product adoption intentions via multiview deep learning. INFORMS J. Comput. 34(1):541–556.Link, Google Scholar
Previous
Back to Top
Next
INFORMS site uses cookies to store information on your computer. Some are essential to make our site work; Others help us improve the user experience. By using this site, you consent to the placement of these cookies. Please read our Privacy Statement to learn more.
Agree