Tutorial: Extracting Unstructured Text Using Large Language Models

References

• Atıl B, Aykent S, Chittams A, Fu L, Passonneau RJ, Radcliffe E, Rajagopal GR, et al. (2025) Non-determinism of “deterministic” LLM system settings in hosted environments. Akter M, Chowdhury T, Eger S, Leiter C, Opitz J, Çano E, eds. Proc. 5th Workshop Evaluation Comparison NLP Systems (Association for Computational Linguistics, Stroudsburg, PA), 135–148.Google Scholar
• Belcak P, Heinrich G, Diao S, Fu Y, Dong X, Muralidharan S, Lin YC, Molchanov P (2025) Small language models are the future of agentic AI. Preprint, submitted June 2, https://arxiv.org/abs/2506.02153.Google Scholar
• Beyene FS, Dancy CL (2026) A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents (Association for Computing Machinery, New York).Google Scholar
• Bray RL (2025) A tutorial on teaching data analytics with generative ai. INFORMS J. Appl. Anal. 55(4):319–343.Link, Google Scholar
• Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, et al. (2020) Language models are few-shot learners. Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H, eds. Advances in Neural Information Processing Systems, vol. 33 (Curran Associates Inc., Red Hook, NY), 1877–1901.Google Scholar
• Bumblauskas D, Igou A, Kalghatgi S, Wetzel C (2022) Public policy and broader applications for the use of text analytics during pandemics. INFORMS J. Appl. Anal. 52(6):568–581.Link, Google Scholar
• Cohen MC, Dai T, Perakis G, Agrawal N, Allon G, Boute RN, Cachon GP, et al. (2026) OM Forum—Supply chain management in the ai era: A vision statement from the operations management community. Manufacturing Service Oper. Management 28(3):687–705.Link, Google Scholar
• Colvin S, Jolibois E, Ramezani H, Garcia Badaracco A, Dorsey T, Montague D, Matveenko S, et al. (2025) Pydantic validation. https://docs.pydantic.dev/latest/.Google Scholar
• De Kok T (2025) ChatGPT for textual analysis? How to use generative LLMs in accounting research. Management Sci. 71(9):7888–7906.Link, Google Scholar
• Fu L, Kuang Z, Song J, Huang M, Yang B, Li Y, Zhu L, et al. (2025) Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. Belgrave D, Zhang C, Lin H, Pascanu R, Koniusz P, Ghassemi M, Chen N, eds. Advances in Neural Information Processing Systems, vol. 38 (Curran Associates Inc, Red Hook, NY), 107930–107981.Google Scholar
• Hansen S, McMahon M, Prat A (2018) Transparency and deliberation within the FOMC: A computational linguistics approach. Quart. J. Econom. 133(2):801–870.Google Scholar
• Humphries M, Leddy LC, Downton Q, Legace M, McConnell J, Murray I, Spence E (2025) Unlocking the archives: Using large language models to transcribe handwritten historical documents. Historical Methods: J. Quant. Interdisciplinary History 58(3):175–193.Google Scholar
• Hutto C, Gilbert E (2014) Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proc. Internat. AAAI Conf. Web Soc. Media 8(1):216–225.Google Scholar
• Kanungo T, Haralick R (1999) An automatic closed-loop methodology for generating character groundtruth for scanned documents. IEEE Trans. Pattern Anal. Machine Intelligence 21(2):179–183.Google Scholar
• Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, Kang J (2019) Biobert: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4):1234–1240.Google Scholar
• Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H, eds. Advances in Neural Information Processing Systems, vol. 33 (Curran Associates Inc., Red Hook, NY), 9459–9474.Google Scholar
• Li X, Culmone V, De Reyck B, Yoo OS (2025) Automating procurement practices using artificial intelligence. INFORMS J. Appl. Anal. 55(3):195–223.Link, Google Scholar
• Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, Liang P (2024) Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics 12:157–173.Google Scholar
• Lu W, Luu RK, Buehler MJ (2025) Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities. NPJ Comput. Materials 11(1):84.Google Scholar
• Markou P, Chan TH (2026) Incentivizing information exchange within groups: The role of voting protocols in us food and drug administration advisory committees. Management Sci. 72(7):5723–5743.Link, Google Scholar
• Mohammadi B (2024) Explaining large language models decisions using Shapley values. Preprint, submitted March 29, https://arxiv.org/abs/2404.01332.Google Scholar
• Neudecker C, Baierer K, Gerber M, Clausner C, Antonacopoulos A, Pletschacher S (2021) A survey of ocr evaluation tools and metrics. Proc. 6th Internat. Workshop Historical Document Imaging Processing (Association for Computing Machinery, New York), 13–18.Google Scholar
• Nguyen TTH, Jatowt A, Coustaty M, Doucet A (2021) Survey of post-ocr processing approaches. ACM Comput. Surv. 54(6):1–37.Google Scholar
• Ouyang S, Zhang JM, Harman M, Wang M (2025) An empirical study of the non-determinism of chatgpt in code generation. ACM Trans. Software Engrg. Methodology 34(2):1–28.Google Scholar
• Renze M (2024) The effect of sampling temperature on problem solving in large language models. Al-Onaizan Y, Bansal M, Chen YN, eds. Proc. Findings Assoc. Comput. Linguistics: EMNLP 2024 (Association for Computational Linguistics, Stroudsburg, PA), 7346–7356.Google Scholar
• Schilling-Wilhelmi M, Ríos-García M, Shabih S, Gil MV, Miret S, Koch CT, Márquez JA, et al. (2025) From text to insight: Large language models for chemical data extraction. Chemical Soc. Rev. 54(3):1125–1150.Google Scholar
• Wasserkrug S (2026) Editorial statement—Harnessing the power of large language models responsibly in applied analytics. INFORMS J. Appl. Anal. 56(4):v–vi.Link, Google Scholar
• Wasserkrug S, Krüger M, A Feldman Y, Shindin E, Zeltyn S (2019) What’s wrong with my dishwasher: Advanced analytics improve the diagnostic process for miele technicians. INFORMS J. Appl. Anal. 49(5):384–396.Link, Google Scholar
• Wu DA (2024) Text-based measure of supply chain risk exposure. Management Sci. 70(7):4781–4801.Link, Google Scholar
• Yang Z, Tang J, Li Z, Wang P, Wan J, Zhong H, Liu X, et al. (2025) CC-OCR: A comprehensive and challenging OCR benchmark for evaluating large multimodal models in literacy. Proc. IEEE/CVF Internat. Conf. Comput. Vision (IEEE/CVF, Los Alamitos, CA), 21744–21754.Google Scholar
• Young RJ, Gillins B, Matthews AM (2025) When models can’t follow: Testing instruction adherence across 256 llms. Preprint, submitted October 18, https://arxiv.org/abs/2510.18892.Google Scholar
• Zhalechian M, Saghafian S, Robles O (2026) Harmonizing safety and speed: A human-algorithm approach to enhance the FDA’s medical device clearance policy. Management Sci., ePub ahead of print August 28, https://doi.org/10.1287/mnsc.2024.06477.Google Scholar
• Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y (2019) Bertscore: Evaluating text generation with BERT. Preprint, submitted April 21, https://arxiv.org/abs/1904.09675.Google Scholar
• Zhu Y, Sen S, Everhart A, Karaca-Mandic P (2025) A deep learning approach for predicting FDA’s 510(k) medical device recalls using device citation relationships. Inform. Systems Res. 0(0):1–24. pre-print.Google Scholar