Reciprocal Human-Machine Learning: A Theory and an Instantiation for the Case of Message Classification

References

• Abbasi A, Chen H (2008) CyberGate: A design framework and system for text analysis of computer-mediated communication. Management Inform. Systems Quart. 32(4):811–837.Crossref, Google Scholar
• Abbasi A, Zhou Y, Deng S, Zhang P (2018) Text analytics to support sense-making in social media: A language-action perspective. Management Inform. Systems Quart. 42(2):427–464.Crossref, Google Scholar
• Abdel-Karim R, Reda Y, Abdel-Fattah A (2020) Nanostructured materials-based nanosensors. J. Electrochemical Soc. 167(3):037554.Crossref, Google Scholar
• Adam MT, Gregor S, Hevner A, Morana S (2021) Transactions on Human-computer interaction. AIS Trans. Human-Comput. Interactions 13(1):1–11.Google Scholar
• Amir O, Doshi-Velez F, Sarne D (2019) Summarizing agent strategies. Autonomic Agent Multi Agent Systems 33(5):628–644.Crossref, Google Scholar
• Ansari F, Erol S, Sihn W (2018) Rethinking human-machine learning in industry 4.0: How does the paradigm shift treat the role of human learning? Procedia Manufacturing 23:117–122.Crossref, Google Scholar
• Baird A, Maruping LM (2021) The next generation of research on IS use: A theoretical framework of delegation to and from agentic IS artifacts. Management Inform. Systems Quart. 45(1):315–341.Crossref, Google Scholar
• Baralou E, Tsoukas H (2015) How is new organizational knowledge created in a virtual context? An ethnographic study. Organ. Stud. 36(5):593–620.Crossref, Google Scholar
• Bazeley P, Jackson K (2019) Qualitative Data Analysis with NVivo (SAGE Publications, London).Google Scholar
• Boland RJ, Tenkasi RV, Te’eni D (1994) Designing information technology to support distributed cognition. Organ. Sci. 5(3):456–475.Link, Google Scholar
• Budd S, Robinson EC, Kainz B (2021) A survey on active learning and human-in-the-loop deep learning for medical image analysis. Medical Image Anal. 71:102062.Crossref, Google Scholar
• Carvalho DV, Pereira EM, Cardoso JS (2019) Machine learning interpretability: A survey on methods and metrics. Electronics (Basel) 8(8):832.Crossref, Google Scholar
• Chang W, Cheng J, Allaire JJ, Xie Y, McPherson J (2021) shiny: Web Application Framework for R. R package version 1.6.0. https://CRAN.R-project.org/package=shiny.Google Scholar
• Creswell J, Creswell J (2017) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (Sage Publications, London).Google Scholar
• Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, submitted October 11, https://arxiv.org/abs/1810.04805.Google Scholar
• Diedrich A, Eriksson-Zetterquist U, Styhre A (2011) Sorting people out: The uses of one-dimensional classificatory schemes in a multi-dimensional world. Cultural Organ. 17(4):271–292.Crossref, Google Scholar
• Duarte N, Llanso E, Loup AC (2018) Mixed messages? The limits of automated social media content analysis. Proc. 1st Conf. on Fairness, Accountability and Transparency (Center for Democracy and Technology, Washington, DC), 106.Google Scholar
• Enarsson T, Enqvist L, Naarttijärvi M (2021) Approaching the human in the loop: Legal perspectives on hybrid human/algorithmic decision-making in three contexts. Inform. Comm. Tech. Law 31(1):123–153.Crossref, Google Scholar
• Fitts PM (1951) Human Engineering for an Effective Air-Navigation and Traffic-Control System (NRC Committee on Aviation Psychology).Google Scholar
• Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J. Statist. Software 33(1):1.Crossref, Google Scholar
• Fromkin V, Rodman R, Hyams N (2011) An Introduction to Language, 9th ed. (Wadsworth, Boston).Google Scholar
• Fügener A, Grahl J, Gupta A, Ketter W (2021) Will humans-in-the-loop become borgs? Merits and pitfalls of working with AI. Management Inform. Systems Quart. 45(3):1527–1556.Crossref, Google Scholar
• Gardner M, Grus J, Neumann M, Tafjord O, Dasigi P, Liu N, Peters M, et al. (2018) Allennlp: A deep semantic natural language processing platform. Preprint, submitted May 31, https://arxiv.org/abs/1803.07640.Google Scholar
• Germonprez M, Kendall JE, Kendall KE, Mathiassen L, Young B, Warner B (2017) A theory of responsive design: A field study of corporate engagement with open source communities. Inform. Systems Res. 28(1):64–83.Link, Google Scholar
• Gillies M, Fiebrink R, Tanaka A, Garcia J, Bevilacqua F, Heloir A, Nunnari F, et al. (2016) Human-centered machine learning. Proc. CHI Conf. Extended Abstracts on Human Factors in Comput. Systems (ACM, New York), 3558–3565.Google Scholar
• Goldstein EB (2014) Cognitive Psychology: Connecting Mind, Research and Everyday Experience (Nelson Education, Ontario).Google Scholar
• Grbich C (2012) Qualitative Data Analysis: An Introduction (Sage, London).Google Scholar
• Gregor S, Hevner AR (2013) Positioning and presenting design science research for maximum impact. Management Inform. Systems Quart. 37(2):337–355.Crossref, Google Scholar
• Groh M (2022) Identifying the context shift between test benchmarks and production data. Preprint, submitted September 22, https://arxiv.org/abs/2207.01059.Google Scholar
• Grønsund T, Aanestad M (2020) Augmenting the algorithm: Emerging human-in-the-loop work configurations. J. Strategic Inform. Systems 29(2):101614.Crossref, Google Scholar
• Hevner A, March S, Park J, Ram S (2004) Design science research in information systems. Management Inform. Systems Quart. 28(1):75–105.Crossref, Google Scholar
• Holzer E, Kent O (2013) A Philosophy of Havruta: Understanding and Teaching the Art of Text Study in Pairs (Academic Studies Press, Boston).Crossref, Google Scholar
• Holzinger A, Weippl E, Tjoa AM, Kieseberg P (2021) Digital transformation for sustainable development goals (SDGs) – A security, safety and privacy perspective on AI. Nugent R, ed. Proc. Internat. Cross-Domain Conf. for Machine Learn. and Knowledge Extraction, Part of Lecture Notes in Computer Science (Springer, Cham, Switzerland), 1–20.Google Scholar
• Iivari J (2015) Distinguishing and contrasting two strategies for design science research. Eur. J. Inform. Systems 24(1):107–115.Crossref, Google Scholar
• Ip W, Damodaran L, Olphert CW, Maguire MC (1990) The use of task allocation charts in system design: A critical appraisal. Diaper D, Gilmore DJ, eds. Proc. IFIP TC13 3rd Internat. Conf. on Human-Computer Interaction (North-Holland Publishing, Amsterdam), 289–294.Google Scholar
• Ji S, Pan S, Cambria E, Marttinen P, Philip SY (2021) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans. Neural Networks Learn. Systems 33(2):494–514.Crossref, Google Scholar
• Jörg T (2004) A theory of reciprocal learning in dyads. Cognitive Systems 6(2/3):159–170.Google Scholar
• Jörg T (2009) Thinking in complexity about learning and education: A programmatic view. Complicity 6(1):22.Crossref, Google Scholar
• Katz A, Te’eni D (2007) The contingent impact of contextualization on computer-mediated collaboration. Organ. Sci. 18(2):261–279.Link, Google Scholar
• Kent O (2010) A theory of Havruta learning. J. Jewish Ed. 76(3):215–245.Crossref, Google Scholar
• Khashabi D, Azer ES, Khot T, Sabharwal A, Roth D (2020) On the possibilities and limitations of multi-hop reasoning under linguistic imperfections. Preprint, submitted May 1, https://arxiv.org/abs/1901.02522.Google Scholar
• Lebovitz S, Levina N, Lifshitz-Assaf H (2021) Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts’ know-what. Management Inform. Systems Quart. 45(3):1501–1526.Crossref, Google Scholar
• Lee AS, Thomas M, Baskerville RL (2015) Going back to basics in design science: From the information technology artifact to the information systems artifact. Inform. Systems J. 25(1):5–21.Crossref, Google Scholar
• Lipton ZC (2018) The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16(3):31–57.Crossref, Google Scholar
• Marcellino W, Johnson C, Posard MN, Helmus TC (2020) Foreign Interference in the 2020 Election: Tools for Detecting Online Election Interference (RAND Corporation, Santa Monica, CA).Crossref, Google Scholar
• McKinney SM, Sieniek M, Godbole V, Godwin J, et al. (2020) International evaluation of an AI system for breast cancer screening. Nature 577(7788):89–94.Crossref, Google Scholar
• McTavish DG, Pirro EB (1990) Contextual content analysis. Qual. Quant. 24(3):245–265.Crossref, Google Scholar
• Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. Preprint, submitted September 7, https://arxiv.org/abs/1301.3781.Google Scholar
• Mokryn O, Ben-Shoshan H (2021) Domain-based latent personal analysis and its use for impersonation detection in social media. User Modeling User-Adaptation Interaction 31(4):785–828.Crossref, Google Scholar
• Moreno R (2004) Decreasing cognitive load for novice students: Effects of explanatory vs. corrective feedback in discovery-based multimedia. Instrument Sci. 32(1):99–113.Google Scholar
• Nonaka I, Takeuchi H (1995) The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation (Oxford University Press, Oxford, UK).Crossref, Google Scholar
• Nonaka I, Toyama R, Nagata A (2000) A firm as a knowledge-creating entity: A new perspective on the theory of the firm. Industrial Corporate Change 9(1):1–20.Crossref, Google Scholar
• Novak JD (2010) Learning, Creating, and Using Knowledge: Concept Maps as Facilitative Tools in Schools and Corporations (Routledge, London).Crossref, Google Scholar
• Nunamaker JF Jr, Twyman NW, Giboney JS, Briggs RO (2017) Creating high-value real-world impact through systematic programs of research. Management Inform. Systems Quart. 41(2):335–351.Crossref, Google Scholar
• Omohundro S (2014) Autonomous technology and the greater human good. J. Experiment. Theoretical Artificial Intelligence 26(3):303–315.Crossref, Google Scholar
• Pang B, Lee L (2004) A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Preprint, submitted September 29, https://arxiv.org/abs/cs/0409058.Google Scholar
• Pearl J (2019) The seven tools of causal inference, with reflections on machine learning. Comm. ACM 62(3):54–60.Crossref, Google Scholar
• R Core Team (2021) R: A language and environment for statistical computing. https://www.R-project.org/.Google Scholar
• Raisch S, Krakowski S (2020) Artificial intelligence and management: The automation–augmentation paradox. Acad. Management Rev. 46(1):192–210.Crossref, Google Scholar
• Ribeiro MT, Singh S, Guestrin C (2016) “Why should I trust you?” Explaining the predictions of any classifier. Krishnapuram B, Shah M, eds. Proc. 22nd ACM SIGKDD Internat. Conf. on Knowledge Discovery and Data Mining (ACM, New York), 1135–1144.Google Scholar
• Riessman CK (2011) What’s different about narrative inquiry? Cases, categories and contexts. Silverman D, ed. Qualitative Research: Issues of Theory, Method, and Practice, 3rd ed. (Sage, London), 310–330.Google Scholar
• Roth WM (2005) Making classifications (at) work: Ordering practices in science. Soc. Stud. Sci. 35(4):581–621.Crossref, Google Scholar
• Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Natural Machine Intelligence 1(5):206–215.Crossref, Google Scholar
• Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, et al. (2015) Imagenet large scale visual recognition challenge. Internat. J. Comput. Vision 115:211–252.Crossref, Google Scholar
• Sapir A, Drori I, Ellis S (2016) The practices of knowledge creation: Collaboration between peripheral and core occupational communities. Eur. Managment Rev. 13(1):19–36.Crossref, Google Scholar
• Schreier M (2012) Qualitative Content Analysis in Practice (Sage Publications, London).Crossref, Google Scholar
• Schwartz DG (1995) Cooperating Heterogeneous Systems (Kluwer Academic Publishers, Alphen aan den Rijn, Netherlands).Crossref, Google Scholar
• Schwartz DG, Te’eni D (2000) Tying knowledge to action with kMail. IEEE Intelligent Systems Their Appl. 15(3):33–39.Crossref, Google Scholar
• Schwartz DG, Yahav I (2021) Knowledge contribution diagrams for design science research: A novel graphical technique. Proc. Internat. Conf. on Design Sci. Res. in Inform. Systems and Tech., Part of the Lecture Notes in Computer Science (Springer, Berlin), 174–187.Google Scholar
• Seidel S, Berente N, Lindberg A, Lyytinen K, Nickerson JV (2018) Autonomous tools and design: A triple-loop approach to human-machine learning. Comm. ACM 62(1):50–57.Crossref, Google Scholar
• Sein MK, Henfridsson O, Purao S, Rossi M, Lindgren R (2011) Action design research. Management Inform. Systems Quart. 35(1):37–56.Crossref, Google Scholar
• Sengupta K, Te’eni D (1993) Cognitive feedback in GDSS: Improving control and convergence. Management Inform. Systems Quart. 17(1):87–113.Crossref, Google Scholar
• Shrestha YR, Ben-Menahem SM, Von Krogh G (2019) Organizational decision-making structures in the age of artificial intelligence. California Management Rev. 61(4):66–83.Crossref, Google Scholar
• Silverman G, Sommer U (2019) Prevalent sentiments of the concept of jihad in the public commentsphere. Stud. Conflict Terrorism 45(7):579–607.Crossref, Google Scholar
• So C (2020a) Human-in-the-loop design cycles: A process framework that integrates design sprints, agile processes, and machine learning with humans. Proc. 1st Internat. Conf. on Artificial Intelligence in HCI, AI-HCI, Part of the Lecture Notes in Computer Science (Springer, Berlin), 136–145.Google Scholar
• So C (2020b) Understanding the prediction mechanism of sentiments by XAI visualization. Proc. 4th Internat. Conf. on Natural Language Processing and Inform. Retrieval (ACM, New York), 75–80.Google Scholar
• Son JY, Goldstone RL (2009) Contextualization in perspective. Cognitive Instruction 27(1):51–89.Crossref, Google Scholar
• Sturm T, Gerlach JP, Pumplun L, Mesbah N, Peters F, Tauchert C, Nan N, et al. (2021) Coordinating human and machine learning for effective organizational learning. Management Inform. Systems Quart. 45(3):1581–1602.Crossref, Google Scholar
• Suchman LA (1987) Plans and Situated Actions: The Problem of Human-Machine Communication (Cambridge University Press, Cambridge, UK).Google Scholar
• Suchman LA (2007) Human-Machine Reconfigurations: Plans and Situated Actions (Cambridge University Press, Cambridge, UK).Google Scholar
• Tausczik YR, Pennebaker JW (2009) The psychological meaning of words: LIWC and computerized text analysis methods. J. Language Soc. Psych. 29(1):24–54.Crossref, Google Scholar
• Te’eni D (2001) Review: A cognitive-affective model of organizational communication for designing IT. Management Inform. Systems Quart. 25(2):251–312.Crossref, Google Scholar
• Van den Broek E, Sergeeva A, Huysman M (2021) When the machine meets the expert: An ethnography of developing AI for hiring. Management Inform. Systems Quart. 45(3):1557–1580.Crossref, Google Scholar
• Vanides J, Yin Y, Tomita M, Ruiz-Primo MA (2005) Concept maps. Sci. Scope 28(8):27–31.Google Scholar
• Vimalkumar M, Gupta A, Sharma D, Dwivedi Y (2021) Understanding the effect that task complexity has on automation potential and opacity: Implications for algorithmic fairness. AIS Trans. Human-Comput. Interactions 13(1):104–129.Crossref, Google Scholar
• Vygotsky LS, Rice E (1978) Mind in Society: The Development of Higher Mental Processes (Harvard University Press, Cambridge, MA).Google Scholar
• Wang X, Kapanipathi P, Musa R, Yu M, Talamadupula K, Abdelaziz I, Chang M, et al. (2019) Improving natural language inference using external knowledge in the science questions domain. Proc. Conf. AAAI Artificial Intelligence 33(1):7208–7215.Crossref, Google Scholar
• Weick KE, Sutcliffe KM, Obstfeld D (2005) Organizing and the process of sensemaking. Organ. Sci. 16(4):409–421.Link, Google Scholar
• Woods DD, Hollnagel E (2006) Joint Cognitive Systems: Patterns in Cognitive Systems Engineering (CRC Press, Boca Raton, FL).Crossref, Google Scholar
• Zagalsky A, Te’eni D, Yahav I, Schwartz DG, Silverman G, Cohen D, Mann Y, Lewinsky D (2021) The design of reciprocal learning between human and artificial intelligence. Nichos J, ed. Proc. ACM Human-Comput. Interaction, 5(CSCW2) (Association for Computing Machinery, New York), 1–36.Google Scholar
• Zamani ED, Griva A, Spanaki K, O’Raghallaigh P, Sammon D (2021) Making sense of business analytics in project selection and prioritisation: Insights from the start-up trenches. Inform. Tech. People.Crossref, Google Scholar
• Zerilli J, Knott A, Maclaurin J, Gavaghan C (2019) Algorithmic decision-making and the control problem. Minds Machine 29(4):555–578.Crossref, Google Scholar
• Zhao R, Mao K (2018) Fuzzy bag-of-words model for document representation. IEEE Trans. Fuzzy Systems 26(2):794–804.Crossref, Google Scholar