Collaborative Intelligence in Sequential Experiments: A Human-in-the-Loop Framework for Drug Discovery

References

• Abbasi A, Parsons J, Pant G, Sheng ORL, Sarker S (2024) Pathways for design research on artificial intelligence. Inform. Systems Res. 35(2):441–459.Link, Google Scholar
• Adjeroh D, Allaga M, Tan J, Lin J, Jiang Y, Abbasi A, Zhou X (2018) Feature-based and string-based models for predicting RNA-protein interaction. Molecules 23(3):697.Crossref, Google Scholar
• Amershi S, Cakmak M, Knox WB, Kulesza T (2014) Power to the people: The role of humans in interactive machine learning. AI Magazine 35(4):105–120.Crossref, Google Scholar
• Balakrishnan M, Ferreira K, Tong J (2022) Improving human-algorithm collaboration: Causes and mitigation of over-and under-adherence. Preprint, submitted December 21, https://doi.org/10.2139/ssrn.4298669.Google Scholar
• Bernard J, Hutter M, Zeppelzauer M, Fellner D, Sedlmair M (2017) Comparing visual-interactive labeling with active learning: An experimental study. IEEE Trans. Visualization Comput. Graphics 24(1):298–308.Crossref, Google Scholar
• Bernard J, Zeppelzauer M, Lehmann M, Müller M, Sedlmair M (2018) Towards user-centered active learning algorithms. Comput. Graphics Forum 37(3):121–132.Crossref, Google Scholar
• Boyacı T, Canyakmaz C, De Véricourt F (2024) Human and machine: The impact of machine input on decision making under cognitive limitations. Management Sci. 70(2):1258–1275.Link, Google Scholar
• Cai C, Wang S, Xu Y, Zhang W, Tang K, Ouyang Q, Lai L, Pei J (2020) Transfer learning for drug discovery. J. Medicinal Chemistry 63(16):8683–8694.Crossref, Google Scholar
• Chen Q, Ho YJI, Sun P, Wang D (2024) The philosopher’s stone for science—The catalyst change of AI for scientific creativity. Proc. Pacific Asia Conf. Inform. Systems (PACIS 2024) (Association for Information Systems, Atlanta), 1770.Google Scholar
• Das P, Sercu T, Wadhawan K, Padhi I, Gehrmann S, Cipcigan F, Chenthamarakshan V, et al. (2021) Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engrg. 5(6):613–623.Crossref, Google Scholar
• Dellermann D, Ebel P, Söllner M, Leimeister JM (2019) Hybrid intelligence. Bus. Inform. Systems Engrg. 61(5):637–643.Crossref, Google Scholar
• Demir M, Cooke NJ, Amazeen PG (2018) A conceptual model of team dynamical behaviors and performance in human-autonomy teaming. Cognitive Systems Res. 52:497–507.Crossref, Google Scholar
• Demir M, McNeese NJ, Cooke NJ (2017) Team situation awareness within the context of human-autonomy teaming. Cognitive Systems Res. 46:3–12.Crossref, Google Scholar
• Deodhar M, Ghosh J, Saar-Tsechansky M, Keshari V (2017) Active learning with multiple localized regression models. INFORMS J. Comput. 29(3):503–522.Link, Google Scholar
• Dietvorst BJ, Simmons JP, Massey C (2018) Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Management Sci. 64(3):1155–1170.Link, Google Scholar
• Donahue K, Chouldechova A, Kenthapadi K (2022) Human-algorithm collaboration: Achieving complementarity and avoiding unfairness. Proc. 2022 ACM Conf. Fairness Accountability Transparency (ACM, New York), 1639–1656. Google Scholar
• Dougherty D, Dunne DD (2012) Digital science and knowledge boundaries in complex innovation. Organ. Sci. 23(5):1467–1484.Link, Google Scholar
• Drews J (2000) Drug discovery: A historical perspective. Science 287(5460):1960–1964.Crossref, Google Scholar
• Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans. Interactive Intelligent Systems 8(2):1–37.Crossref, Google Scholar
• Ekins S, Puhl AC, Zorn KM, Lane TR, Russo DP, Klein JJ, Hickey AJ, Clark AM (2019) Exploiting machine learning for end-to-end drug discovery and development. Nature Materials 18(5):435–441.Crossref, Google Scholar
• Evans JSB (1984) Heuristic and analytic processes in reasoning. British J. Psych. 75(4):451–468.Crossref, Google Scholar
• Evans JA, Foster JG (2011) Metaknowledge. Science 331(6018):721–725.Crossref, Google Scholar
• Evans JSB, Stanovich KE (2013) Dual-process theories of higher cognition: Advancing the debate. Perspect. Psych. Sci. 8(3):223–241.Crossref, Google Scholar
• Fügener A, Grahl J, Gupta A, Ketter W (2021) Will humans-in-the-loop become borgs? Merits and pitfalls of working with AI. MIS Quart. 45(3):1527–1556.Crossref, Google Scholar
• Fügener A, Grahl J, Gupta A, Ketter W (2022) Cognitive challenges in human–artificial intelligence collaboration: Investigating the path toward productive delegation. Inform. Systems Res. 33(2):678–696.Link, Google Scholar
• Gawlikowski J, Tassi CRN, Ali M, Lee J, Humt M, Feng J, Kruspe A, et al. (2023) A survey of uncertainty in deep neural networks. Artificial Intelligence Rev. 56:1513–1589.Crossref, Google Scholar
• Ge R, Zheng Z, Tian X, Liao L (2021) Human–robot interaction: When investors adjust the usage of robo-advisors in peer-to-peer lending. Inform. Systems Res. 32(3):774–785.Link, Google Scholar
• Geva T, Saar-Tsechansky M (2021) Who is a better decision maker? Data-driven expert ranking under unobserved quality. Production Oper. Management 30(1):127–144.Crossref, Google Scholar
• Ghai B, Liao QV, Zhang Y, Bellamy R, Mueller K (2021) Explainable active learning (XAL) toward AI explanations as interfaces for machine teachers. Proc. ACM Human Comput. Interaction, vol. 4 (ACM, New York), 1–28.Crossref, Google Scholar
• Graves A (2011) Practical variational inference for neural networks. Adv. Neural Inform. Processing Systems, vol. 24 (Curran Associates Inc., Red Hook, NY), 2348–2356.Google Scholar
• Guo T, Bardhan IR, Ding Y, Zhang S (2024) An explainable artificial intelligence approach using graph learning to predict intensive care unit length of stay. Inform. Systems Res. 36(3):1478–1501.Google Scholar
• Guo X, Chen G, Wang C, Wei Q, Zhang Z (2021) Calibration of voting-based helpfulness measurement for online reviews: An iterative Bayesian probability approach. INFORMS J. Comput. 33(1):246–261.Link, Google Scholar
• Heer J (2019) Agency plus automation: Designing artificial intelligence into interactive systems. Proc. Natl. Acad. Sci. USA 116(6):1844–1850.Crossref, Google Scholar
• Ho TK, Basu M (2002) Complexity measures of supervised classification problems. IEEE Trans. Pattern Anal. Machine Intelligence 24(3):289–300.Crossref, Google Scholar
• Hollnagel E, Woods DD (2005) Joint Cognitive Systems: Foundations of Cognitive Systems Engineering (CRC Press, Boca Raton).Crossref, Google Scholar
• Holzinger A (2016) Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Inform. 3(2):119–131.Crossref, Google Scholar
• Hüllermeier E, Waegeman W (2021) Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learn. 110(3):457–506.Crossref, Google Scholar
• Ibrahim R, Kim SH, Tong J (2021) Eliciting human judgment for prediction algorithms. Management Sci. 67(4):2314–2325.Link, Google Scholar
• Jain H, Padmanabhan B, Pavlou PA, Raghu T (2021) Editorial for the special section on humans, algorithms, and augmented intelligence: The future of work, organizations, and society. Inform. Systems Res. 32(3):675–687.Link, Google Scholar
• Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, Tunyasuvunakool K, et al. (2021) Highly accurate protein structure prediction with AlphaFold. Nature 596(7873):583–589.Crossref, Google Scholar
• Jussupow E, Spohrer K, Heinzl A, Gawlitza J (2021) Augmenting medical diagnosis decisions? An investigation into physicians’ decision-making process with artificial intelligence. Inform. Systems Res. 32(3):713–735.Link, Google Scholar
• Kahneman D (2011) Thinking, Fast and Slow (Macmillan, New York).Google Scholar
• Kawaguchi K (2021) When will workers follow an algorithm? A field experiment with a retail business. Management Sci. 67(3):1670–1695.Link, Google Scholar
• Kim S, Mai TD, Han S, Park S, Nguyen DT, So J, Singh K, Cha M (2022) Active learning for human-in-the-loop customs inspection. IEEE Trans. Knowledge Data Engrg. 35(12):12039–12052.Crossref, Google Scholar
• Kitchens B, Claggett J, Abbasi A (2024) Timely, granular, and actionable: Designing a social listening platform for public health 3.0. MIS Quart. 48(3):899–930.Crossref, Google Scholar
• Lakshminarayanan B, Pritzel A, Blundell C (2017) Simple and scalable predictive uncertainty estimation using deep ensembles. Adv. Neural Inform. Processing Systems, vol. 30 (Curran Associates Inc., Red Hook, NY), 6405–6416.Google Scholar
• Lattimore T, Szepesvári C (2020) Bandit Algorithms (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Lebovitz S (2019) Diagnostic doubt and artificial intelligence: An inductive field study of radiology work. ICIS 2019 Proc. (New York University, New York), 11.Google Scholar
• Lebovitz S, Levina N, Lifshitz-Assaf H (2021) Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts’ know-what. MIS Quart. 45(3):1501–1526.Crossref, Google Scholar
• Lee K, Ram S (2024) Explainable deep learning for false information identification: An argumentation theory approach. Inform. Systems Res. 35(2):890–907.Link, Google Scholar
• Lewis DD (1995) A sequential algorithm for training text classifiers: Corrigendum and additional data. ACM SIGIR Forum 29(2):13–19.Crossref, Google Scholar
• Li Y, De-Arteaga M, Saar-Tsechansky M (2022) When more data lead us astray: Active data acquisition in the presence of label bias. Proc. AAAI Conf. Human Comput. Crowdsourcing, 133–146.Google Scholar
• Li J, Larsen K, Abbasi A (2020) TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning. MIS Quart. 44(4):1733–1772.Crossref, Google Scholar
• Li Y, Reyes KG, Vazquez-Anderson J, Wang Y, Contreras LM, Powell WB (2018) A knowledge gradient policy for sequencing experiments to identify the structure of RNA molecules using a sparse additive belief model. INFORMS J. Comput. 30(4):750–767.Link, Google Scholar
• Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, Smetanin N, et al. (2023) Evolutionary-scale prediction of atomic-level protein structure with a language model. Science 379(6637):1123–1130.Crossref, Google Scholar
• Lindauer M, Karl F, Klier A, Moosbauer J, Tornede A, Mueller AC, Hutter F, Feurer M, Bischl B (2024) Position: A call to action for a human-centered AutoML paradigm. Proc. 41st Internat. Conf. Machine Learn., vol. 235 (PMLR, New York), 30566–30584.Google Scholar
• Liu R, Li J, Zivkovic M, Abbasi A (2025) Automating in high-expertise, low-label environments: Evidence-based medicine by expert-augmented few-shot learning. MIS Quart. 49(3):1049–1094.Crossref, Google Scholar
• Lorena AC, Garcia LP, Lehmann J, Souto MC, Ho TK (2019) How complex is your classification problem? A survey on measuring classification complexity. ACM Comput. Surveys 52(5):1–34.Crossref, Google Scholar
• Lou B, Wu L (2021) AI on drugs: Can artificial intelligence accelerate drug development? Evidence from a large-scale examination of bio-pharma firms. MIS Quart. 45(3):1451–1482.Crossref, Google Scholar
• Lu T, Zhang Y (2025) 1+ 1 > 2? Information, humans, and machines. Inform. Systems Res. 36(1):394–418.Link, Google Scholar
• Madni AM, Madni CC (2018) Architectural framework for exploring adaptive human-machine teaming options in simulated dynamic environments. Systems 6(4):44.Crossref, Google Scholar
• McCormack J, Hutchings P, Gifford T, Yee-King M, Llano MT, D’inverno M (2020) Design considerations for real-time collaboration with creative artificial intelligence. Organised Sound 25(1):41–52.Crossref, Google Scholar
• Menchik DA (2014) Decisions about knowledge in medical practice: The effect of temporal features of a task. Amer. J. Sociol. 120(3):701–749.Crossref, Google Scholar
• Mengis J, Nicolini D, Swan J (2018) Integrating knowledge in the face of epistemic uncertainty: Dialogically drawing distinctions. Management Learn. 49(5):595–612.Crossref, Google Scholar
• Mosqueira-Rey E, Hernández-Pereira E, Alonso-Ríos D, Bobes-Bascarán J, Fernández-Leal Á (2023) Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Rev. 56(4):3005–3054.Crossref, Google Scholar
• Mou Y, Xu K (2017) The media inequality: Comparing the initial human-human and human-AI social interactions. Comput. Human Behav. 72:432–440.Crossref, Google Scholar
• Mouchlis VD, Afantitis A, Serra A, Fratello M, Papadiamantis AG, Aidinis V, Lynch I, Greco D, Melagraki G (2021) Advances in de novo drug design: From conventional to machine learning methods. Internat. J. Molecular Sci. 22(4):1676.Crossref, Google Scholar
• Nahal Y, Menke J, Martinelli J, Heinonen M, Kabeshov M, Janet JP, Nittinger E, Engkvist O, Kaski S (2024) Human-in-the-loop active learning for goal-oriented molecule generation. J. Cheminformatics 16(1):1–24.Crossref, Google Scholar
• Nasrollahzadeh AA, Khademi A (2022) Dynamic programming for response-adaptive dose-finding clinical trials. INFORMS J. Comput. 34(2):1176–1190.Link, Google Scholar
• Negoescu DM, Frazier PI, Powell WB (2011) The knowledge-gradient algorithm for sequencing experiments in drug discovery. INFORMS J. Comput. 23(3):346–363.Link, Google Scholar
• Northrup J (2005) The pharmaceutical sector, Chapter 2. The Business of Healthcare Innovation (Cambridge University Press, Cambridge, UK), 27–102Crossref, Google Scholar
• Padmanabhan B, Fang X, Sahoo N, Burton-Jones A (2022) Machine learning in information systems research. MIS Quart. 46(1):iii–xix.Google Scholar
• Petricek T, van Den Burg GJ, Nazábal A, Ceritli T, Jiménez-Ruiz E, Williams CK (2022) AI assistants: A framework for semi-automated data wrangling. IEEE Trans. Knowledge Data Engrg. 35(9):9295–9306.Crossref, Google Scholar
• Raees M, Meijerink I, Lykourentzou I, Khan VJ, Papangelis K (2024) From explainable to interactive AI: A literature review on current trends in human-AI interaction. Internat. J. Human Comput. Stud. 189:103301.Crossref, Google Scholar
• Rafailov R, Sharma A, Mitchell E, Manning CD, Ermon S, Finn C (2024) Direct preference optimization: Your language model is secretly a reward model. Adv. Neural Inform. Processing Systems, vol. 36 (Curran Associates Inc., Red Hook, NY), 53728–53741.Google Scholar
• Rai A, Burton-Jones A, Chen H, Gupta A, Hevner AR, Ketter W, Parsons J, Rao HR, Sarkar S, Yoo Y (2017) Editor’s comments: Diversity of design science research.Google Scholar
• Ravina E (2011) The Evolution of Drug Discovery: From Traditional Medicines to Modern Drugs (John Wiley & Sons, Hoboken, NJ).Google Scholar
• Réda C, Kaufmann E, Delahaye-Duriez A (2020) Machine learning applications in drug development. Comput. Structural Biotechnology J. 18:241–252.Crossref, Google Scholar
• Reker D, Schneider G (2015) Active-learning strategies in computer-assisted drug discovery. Drug Discovery Today 20(4):458–465.Crossref, Google Scholar
• Riedl MO (2019) Human-centered artificial intelligence and machine learning. Human Behav. Emerging Tech. 1(1):33–36.Crossref, Google Scholar
• Ringel MS, Scannell JW, Baedeker M, Schulze U (2020) Breaking Eroom’s law. Nature Rev. Drug Discovery 19(12):833–835.Crossref, Google Scholar
• Saar-Tsechansky M, Provost F (2007) Decision-centric active learning of binary-outcome models. Inform. Systems Res. 18(1):4–22.Link, Google Scholar
• Saar-Tsechansky M, Melville P, Provost F (2009) Active feature-value acquisition. Management Sci. 55(4):664–684.Link, Google Scholar
• Schanke S, Burtch G, Ray G (2021) Estimating the impact of “humanizing” customer service chatbots. Inform. Systems Res. 32(3):736–751.Link, Google Scholar
• Schneider G (2018) Automating drug discovery. Nature Rev. Drug Discovery 17(2):97–113.Crossref, Google Scholar
• Settles B (2009) Active Learning Literature Survey.Google Scholar
• Shin D (2024) Misinformation and algorithmic bias, Chapter 2. Artificial Misinformation: Exploring Human-Algorithm Interaction Online (Springer, Berlin), 15–47.Crossref, Google Scholar
• Shively RJ, Lachter J, Brandt SL, Matessa M, Battiste V, Johnson WW (2017) Why human-autonomy teaming? Internat. Conf. Appl. Human Factors Ergonomics (Springer, New York), 3–11.Google Scholar
• Sinha AP, Zhao H (2008) Incorporating domain knowledge into data mining classifiers: An application in indirect lending. Decision Support Systems 46(1):287–299.Crossref, Google Scholar
• Slivkins A (2011) Contextual bandits with similarity information. Proc. 24th Annual Conf. Learn. Theory (PMLR, Cambridge, MA), 679–702.Google Scholar
• Sturm T, Gerlach JP, Pumplun L, Mesbah N, Peters F, Tauchert C, Nan N, Buxmann P (2021) Coordinating human and machine learning for effective organizational learning. MIS Quart. 45(3):1581–1602.Crossref, Google Scholar
• Sun J, Zhang DJ, Hu H, Van Mieghem JA (2022) Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations. Management Sci. 68(2):846–865.Link, Google Scholar
• Sundin I, Voronov A, Xiao H, Papadopoulos K, Bjerrum EJ, Heinonen M, Patronov A, Kaski S, Engkvist O (2022) Human-in-the-loop assisted de novo molecular design. J. Cheminformatics 14(1):86.Crossref, Google Scholar
• Svensson HG, Bjerrum EJ, Tyrchan C, Engkvist O, Chehreghani MH (2022) Autonomous drug design with multi-armed bandits. 2022 IEEE Internat. Conf. Big Data (IEEE Computer Society, Washington, DC), 5584–5592.Google Scholar
• Te’eni D, Yahav I, Zagalsky A, Schwartz D, Silverman G, Cohen D, Mann Y, Lewinsky D (2023) Reciprocal human-machine learning: A theory and an instantiation for the case of message classification. Management Sci., ePub ahead of print November 14, https://doi.org/10.1287/mnsc.2022.03518.Link, Google Scholar
• Vamathevan J, Clark D, Czodrowski P, Dunham I, Ferran E, Lee G, Li B, et al. (2019) Applications of machine learning in drug discovery and development. Nature Rev. Drug Discovery 18(6):463–477.Crossref, Google Scholar
• Van den Broek E, Sergeeva A, Huysman M (2021) When the machine meets the expert: An ethnography of developing AI for hiring. MIS Quart. 45(3):1557–1580.Crossref, Google Scholar
• Wang J, Ipeirotis PG, Provost F (2017) Cost-effective quality assurance in crowd labeling. Inform. Systems Res. 28(1):137–158.Link, Google Scholar
• Wang W, Yang M, Sun T (2023) Human-AI co-creation in product ideation: The dual view of quality and diversity. Preprint, submitted December 20, https://doi.org/10.2139/ssrn.4668241.Google Scholar
• Xu W, Dainoff MJ, Ge L, Gao Z (2023) Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI. Internat. J. Human Comput. Interaction 39(3):494–518.Crossref, Google Scholar
• Yin J, Luo J, Brown SA (2021) Learning from crowdsourced multi-labeling: A variational Bayesian approach. Inform. Systems Res. 32(3):752–773.Abstract, Google Scholar
• Zhang J, Fiers P, Witte KA, Jackson RW, Poggensee KL, Atkeson CG, Collins SH (2017) Human-in-the-loop optimization of exoskeleton assistance during walking. Science 356(6344):1280–1284.Crossref, Google Scholar
• Zheng Z, Padmanabhan B (2006) Selectively acquiring customer information: A new data acquisition problem and an active learning-based solution. Management Sci. 52(5):697–712.Link, Google Scholar
• Zheng N, Liu Z, Ren P, Ma Y, Chen S, Yu S, Xue J, Chen B, Wang F (2017) Hybrid-augmented intelligence: Collaboration and cognition. Frontiers Inform. Tech. Electronic Engrg. 18(2):153–179.Crossref, Google Scholar
• Zhou Z, Wang Y, Mamani H, Coffey DG (2019) How do tumor cytogenetics inform cancer treatments? Dynamic risk stratification and precision medicine using multi-armed bandits. Preprint, submitted July 12, https://doi.org/10.2139/ssrn.3405082.Google Scholar
• Zhou T, Wang Y, Yan L, Tan Y (2023) Spoiled for choice? Personalized recommendation for healthcare decisions: A multiarmed bandit approach. Inform. Systems Res. 34(4):1493–1512.Link, Google Scholar
• Zoffmann S, Vercruysse M, Benmansour F, Maunz A, Wolf L, Blum Marti R, Heckel T, et al. (2019) Machine learning-powered antibiotics phenotypic drug discovery. Sci. Rep. 9(1):5013.Crossref, Google Scholar