Reducing Manual Labeling Effort in Imbalanced Data Sets: Active Learning for Detecting Illicit Massage Business Reviews

References

• Alvari H, Shakarian P, Snyder JEK (2017) Semi-supervised learning for detecting human trafficking. Security Informatics 6(1):1–14.Crossref, Google Scholar
• Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34(6):26–38.Crossref, Google Scholar
• Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Machine Learn. 15(2):201–221.Crossref, Google Scholar
• de Vries I, Radford J (2022) Identifying online risk markers of hard-to-observe crimes through semi-inductive triangulation: The case of human trafficking in the United States. British J. Criminology 62(3):639–658.Crossref, Google Scholar
• Demand Abolition (2018) Who buys sex? Understanding and disrupting illicit market demand. Accessed July 27, 2022, https://www.demandabolition.org/wp-content/uploads/2019/07/Demand-Buyer-Report-July-2019.pdf.Google Scholar
• Diaz M, Panangadan A (2020) Natural language-based integration of online review datasets for identification of sex trafficking businesses. Proc. 2020 IEEE 21st Internat. Conf. Inform. Reuse Integration Data Sci. (IEEE Press, Piscataway, NJ), 259–264.Google Scholar
• Dubrawski A, Miller K, Barnes M, Boecking B, Kennedy E (2015) Leveraging publicly available data to discern patterns of human-trafficking activity. J. Human Trafficking 1(1):65–85.Crossref, Google Scholar
• Esfahani SS, Cafarella MJ, Pouyan MB, DeAngelo GJ, Eneva E, Fano AE (2019) Context-specific language modeling for human trafficking detection from online advertisements. Korhonen A, Traum D, Màrquez L, eds. Proc. 57th Annual Meeting Assoc. Comput. Linguistics (ACL, Florence, Italy), 1180–1184.Google Scholar
• Fang M, Li Y, Cohn T (2017) Learning how to active learn: A deep reinforcement learning approach. Palmer M, Hwa R, Riedel S, eds. Proc. 2017 Conf. Empirical Methods Natural Language Processing (ACL, Copenhagen, Denmark), 595–605.Google Scholar
• Gal Y, Ghahramani Z (2016) Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. Balcan MF, Weinberger KQ, eds. Internat. Conf. Machine Learn. (PMLR, New York), 1050–1059.Google Scholar
• Gal Y, Islam R, Ghahramani Z (2017) Deep Bayesian active learning with image data. Precup D, Teh YW, eds. Internat. Conf. Machine Learn. (PMLR, Sydney, Australia), 1183–1192.Google Scholar
• Global Emancipation Network (2023) Accessed July 9, 2025, https://www.globalemancipation.ngo/whatwedo.Google Scholar
• He J, Chen J, He X, Gao J, Li L, Deng L, Ostendorf M (2016) Deep reinforcement learning with a natural language action space. Erk K, Smith NA, eds. Proc. 54th Annual Meeting Assoc. Comput. Linguistics (ACL, Berlin, Germany), 1621–1630.Google Scholar
• Holub A, Perona P, Burl MC (2008) Entropy-based active learning for object recognition. 2008 IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognition Workshops (IEEE, Anchorage, AK), 1–8.Google Scholar
• Houlsby N, Huszár F, Ghahramani Z, Lengyel M (2011) Bayesian active learning for classification and preference learning. Preprint, submitted December 24, https://arxiv.org/abs/1112.5745.Google Scholar
• Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: A survey. J. Artificial Intelligence Res. 4:237–285.Crossref, Google Scholar
• Keskin B, Bott G, Freeman N (2021) Cracking sex trafficking: Data analysis, pattern recognition, and path prediction. Production Oper. Management 30(4):1110–1135.Crossref, Google Scholar
• Le Q, Mikolov T (2014) Distributed representations of sentences and documents. Xing EP, Jebara T, eds. Internat. Conf. Machine Learn. (PMLR, Bejing, China), 1188–1196.Google Scholar
• Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. Proc. 17th Annual Internat. ACM-SIGIR Conf. Res. Development Inform. Retrieval (Springer-Verlag, Berlin, Heidelberg), 3–12.Google Scholar
• Li R, Tobey M, Mayorga M, Caltagirone S, Özaltın O (2023) Detecting human trafficking: Automated classification of online customer reviews of massage businesses. Manufacturing Service Oper. Management 25(3):1051–1065.Link, Google Scholar
• Li J, Monroe W, Ritter A, Jurafsky D, Galley M, Gao J (2016) Deep reinforcement learning for dialogue generation. Su J, Duh K, Carreras X, eds. Proc. 2016 Conf. Empirical Methods Natural Language Processing (ACL, Austin, TX), 1192–1202.Google Scholar
• MacKay DJ (1992) Information-based objective functions for active data selection. Neural Comput. 4(4):590–604.Crossref, Google Scholar
• Mienye ID, Sun Y (2021) Performance analysis of cost-sensitive learning methods with application to imbalanced medical data. Informatics Medicine Unlocked 25:100690.Crossref, Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, et al. (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533.Crossref, Google Scholar
• Narasimhan K, Yala A, Barzilay R (2016) Improving information extraction by acquiring external evidence with reinforcement learning. Su J, Duh K, Carreras X, eds. Proc. 2016 Conf. Empirical Methods Natural Language Processing (ACL, Austin, TX), 2355–2365.Google Scholar
• Olsson F (2009) A literature survey of active machine learning in the context of natural language processing. Swedish Institute of Computer Science. Accessed October 8, 2023, https://www.diva-portal.org.Google Scholar
• Polaris Project (2019) Human trafficking in illicit massage businesses. Accessed May 28, 2021, https://massagetherapy.nv.gov/Resources/Resource_Home/.Google Scholar
• Polaris Project (2022) Polaris analysis of 2021 data from the national human trafficking hotline. Accessed October 23, 2023, https://polarisproject.org/wp-content/uploads/2020/07/Polaris-Analysis-of-2021-Data-from-the-National-Human-Trafficking-Hotline.pdf.Google Scholar
• Prakash T, Kak AC (2018) Active learning for designing detectors for infrequently occurring objects in wide-area satellite imagery. Comput. Vision Image Understanding 170:92–108.Crossref, Google Scholar
• Ramchandani P, Bastani H, Wyatt E (2025) Unmasking human trafficking risk in commercial sex supply chains with machine learning. Manufacturing Service Oper. Management 27(3):700–719.Google Scholar
• Roy N, McCallum A (2001) Toward optimal active learning through Monte Carlo estimation of error reduction. Internat. Conf. Machine Learn. (Williamstown), 441–448.Google Scholar
• Settles B (2009) Active learning literature survey. University of Wisconsin-Madison Department of Computer Sciences. Accessed October 8, 2023, http://digital.library.wisc.edu/1793/60660.Google Scholar
• Seung HS, Opper M, Sompolinsky H (1992) Query by committee. Proc. Fifth Annual Workshop Comput. Learn. Theory (ACM, New York), 287–294.Google Scholar
• Shen Y, Yun H, Lipton ZC, Kronrod Y, Anandkumar A (2018) Deep active learning for named entity recognition. Blunsom P, Bordes A, Cho K, Cohen S, Dyer C, Grefenstette E, Hermann KM, Rimell L, Weston J, Yih S, eds. Sixth Internat. Conf. Learn. Representations (ACL, Vancouver, Canada), 252–256.Google Scholar
• Shoemark P, Liza FF, Nguyen D, Hale SA, McGillivray B (2019) Room to Glo: A systematic comparison of semantic change detection approaches with word embeddings. Proc. 2019 Conf. Empirical Methods Natural Language Processing 9th Internat. Joint Conf. Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics, Hong Kong, China), 66–76.Google Scholar
• Siddhant A, Lipton ZC (2018) Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. Riloff E, Chiang D, Hockenmaier J, Tsujii J, eds. Proc. 2018 Conf. Empirical Methods Natural Language Processing (ACL, Brussels, Belgium), 2904–2909.Google Scholar
• Sun L, Gong Y (2019) Active learning for image classification: A deep reinforcement learning approach. Second China Sympos. Cognitive Comput. Hybrid Intelligence (IEEE, Xi’an, China), 71–76.Google Scholar
• Sutton RS, Barto AG (2018) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).Google Scholar
• Tong S, Koller D (2001) Support vector machine active learning with applications to text classification. J. Machine Learn. Res. 2:45–66.Google Scholar
• Tong E, Zadeh A, Jones C, Morency L-P (2017) Combating human trafficking with deep multimodal models. Barzilay R, Kan MY, eds. Proc. 55th Annual Meeting Assoc. Comput. Linguistics (ACL, Vancouver, Canada), 1547–1556.Google Scholar
• U.S. Department of Justice (2018) Justice Department leads effort to seize Backpage.com. Accessed October 7, 2021, https://www.justice.gov/opa/pr/justice-department-leads-effort-seize-backpagecom-internet-s-leading-forum-prostitution-ads.Google Scholar
• U.S. Department of State (2021) Trafficking in persons report, 2021. Accessed July 21, 2021, https://www.state.gov/reports/2021-trafficking-in-persons-report/.Google Scholar
• Wang L, Laber E, Saanchi Y, Caltagirone S (2020b) Sex trafficking detection with ordinal regression neural networks. AAAI-20 Workshop on Artificial Intelligence Cyber Security (AAAI Press, New York).Google Scholar
• Wang J, Yan Y, Zhang Y, Cao G, Yang M, Ng MK (2020a) Deep reinforcement active learning for medical image classification. Medical Image Comput. Comput. Assisted Intervention 23rd Internat. Conf. Proc. Part I, vol. 23 (Springer-Verlag, Berlin, Heidelberg), 33–42.Google Scholar
• Yang Y, Loog M (2018) A benchmark and comparison of active learning for logistic regression. Pattern Recognition 83:401–415.Crossref, Google Scholar
• Zhang Y, Lease M, Wallace B (2017) Active discriminative text representation learning. Proc. AAAI Conf. Artificial Intelligence, vol 31, (AAAI Press, San Francisco, CA), 3386–3392.Google Scholar