Mutual Information Surprise: Rethinking Unexpectedness in Autonomous Systems

References

• Ahmed I, Bukkapatnam ST, Botcha B, Ding Y (2025) Toward futuristic autonomous experimentation—A surprise-reacting sequential experiment policy. IEEE Trans. Automation Sci. Engrg. 22:7912–7926.Google Scholar
• Ayed F, Stella L, Januschowski T, Gasthaus J (2020) Anomaly detection at scale: The case for deep distributional time series models. Proc. Internat. Conf. Service-Oriented Comput. (Springer, Berlin, Heidelberg), 97–109.Google Scholar
• Aytekin C, Ni X, Cricri F, Aksu E (2018) Clustering and unsupervised anomaly detection with l-2 normalized deep auto-encoder representations. Proc. Internat. Joint Conf. Neural Networks (IEEE, Piscataway, NJ), 1–6.Google Scholar
• Baldi P (2002) A computational theory of surprise. Blaum M, Farrell PG, eds. Proc. Workshop Honoring Prof. Bob Mceliece on His 60th Birthday: Inform., Coding and Math. (Springer, Berlin, Heidelberg), 1–25.Google Scholar
• Barto A, Mirolli M, Baldassarre G (2013) Novelty or surprise? Frontiers Psych. 4:907.Google Scholar
• Bayram F, Ahmed BS, Kassler A (2022) From concept drift to model degradation: An overview on performance-aware drift detectors. Knowledge Based Systems 245:108632.Google Scholar
• Bickel S, Brückner M, Scheffer T (2009) Discriminative learning under covariate shift. J. Machine Learn. Res. 10(9):2137–2155.Google Scholar
• Bogdoll D, Nitsche M, Zöllner JM (2022) Anomaly detection in autonomous driving: A survey. Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognition (IEEE, Piscataway, NJ), 4487–4498.Google Scholar
• Bondu A, Lemaire V, Boullé M (2010) Exploration vs. exploitation in active learning: A Bayesian approach. Proc. Internat. Joint Conf. Neural Networks (IEEE, Piscataway, NJ), 1–7.Google Scholar
• Burger B, Maffettone PM, Gusev VV, Aitchison CM, Bai Y, Wang X, Li X, et al. (2020) A mobile robotic chemist. Nature 583:237–241.Google Scholar
• Çatal O, Leroux S, De Boom C, Verbelen T, Dhoedt B (2020) Anomaly detection for autonomous guided vehicles using Bayesian surprise. Proc. IEEE/RSJ Internat. Conf. Intelligent Robots Systems (IEEE, Piscataway, NJ), 8148–8153.Google Scholar
• Cebron N, Berthold MR (2009) Active learning for object classification: From exploration to exploitation. Data Mining Knowledge Discovery 18:283–299.Google Scholar
• Chai K (2009) Generalization errors and learning curves for regression with multi-task Gaussian processes. Proc. 23rd Adv. Neural Inform. Processing Systems (ACM, New York), 279–287.Google Scholar
• Chang J, Nikolaev P, Carpena-Núñez J, Rao R, Decker K, Islam AE, Kim J, et al. (2020) Efficient closed-loop maximization of carbon nanotube growth rate using Bayesian optimization. Sci. Rep. 10:9040.Google Scholar
• Cohen K, Zhao Q (2015) Active hypothesis testing for anomaly detection. IEEE Trans. Inform. Theory 61(3):1432–1450.Google Scholar
• Cover TM (1999) Elements of Information Theory (John Wiley & Sons, New York).Google Scholar
• Dai T, Vijayakrishnan S, Szczypiński FT, Ayme JF, Simaei E, Fellowes T, Clowes R, et al. (2024) Autonomous mobile robots for exploratory synthetic chemistry. Nature 635:890–897.Google Scholar
• Dawid AP (1984) Present position and potential developments: Some personal views statistical theory the prequential approach. J. Roy. Statist. Soc.: Ser. A (General) 147(2):278–290.Google Scholar
• Dinparastdjadid A, Supeene I, Engstrom J (2023) Measuring surprise in the wild. Preprint, submitted May 12, https://arxiv.org/abs/2305.07733.Google Scholar
• Doquire G, Verleysen M (2013) Mutual information-based feature selection for multilabel classification. Neurocomputing 122:148–155.Google Scholar
• Faraji M, Preuschoff K, Gerstner W (2018) Balancing new against old information: The role of puzzlement surprise in learning. Neural Comput. 30(1):34–83.Google Scholar
• François D, Wertz V, Verleysen M (2006) The permutation test for feature selection by mutual information. Proc. 14th Eur. Sympos. Artificial Neural Networks (ESANN, Bruges, Belgium), 239–244.Google Scholar
• Friston K, Rigoli F, Ognibene D, Mathys C, Fitzgerald T, Pezzulo G (2015) Active inference and epistemic value. Cognitive Neurosci. 6(4):187–214.Google Scholar
• Hou Y, Chen Z, Wu M, Foo CS, Li X, Shubair RM (2020) Mahalanobis distance based adversarial network for anomaly detection. Proc. IEEE Internat. Conf. Acoustics, Speech and Signal Processing (IEEE, Piscataway, NJ), 3192–3196.Google Scholar
• Islam UJ, Paynabar K, Runger G, Iquebal AS (2025) Dynamic exploration–exploitation trade-off in active learning regression with Bayesian hierarchical modeling. IISE Trans. 57(4):393–407.Google Scholar
• Itti L, Baldi P (2009) Bayesian surprise attracts human attention. Vision Res. 49(10):1295–1306.Google Scholar
• Jin S, Deneault JR, Maruyama B, Ding Y (2022) Autonomous experimentation systems and benefit of surprise-based Bayesian optimization. Proc. Internat. Sympos. Flexible Automation (ASME, Yokohama, Japan), 1–7.Google Scholar
• Joseph VR (2016) Space-filling designs for computer experiments: A review. Quality Engrg. 28(1):28–35.Google Scholar
• Kamenik JF, Szewc M (2023) Null hypothesis test for anomaly detection. Phys. Lett. B 840:137836.Google Scholar
• Kolossa A, Kopp B, Fingscheidt T (2015) A computational analysis of the neural bases of Bayesian inference. Neuroimage 106:222–237.Google Scholar
• Lee MC, Lin JC, Gran EG (2020) Repad: Real-time proactive anomaly detection for time series. Proc. Internat. Conf. Advanced Inform. Networking Appl. (Springer), 1291–1302.Google Scholar
• Leng J, Zhong Y, Lin Z, Xu K, Mourtzis D, Zhou X, Zheng P, Liu Q, Zhao JL, Shen W (2023) Towards resilience in industry 5.0: A decentralized autonomous manufacturing paradigm. J. Manufacturing Systems 71:95–114.Google Scholar
• Levinson J, Askeland J, Becker J, Dolson J, Held D, Kammel S, Kolter JZ, et al. (2011) Towards fully autonomous driving: Systems and algorithms. Proc. IEEE Intelligent Vehicles Sympos (IEEE, Piscataway, NJ), 163–168.Google Scholar
• Liakoni V, Modirshanechi A, Gerstner W, Brea J (2021) Learning in volatile environments with the Bayes factor surprise. Neural Comput. 33(2):269–340.Google Scholar
• Lian B, Kartal Y, Lewis FL, Mikulski DG, Hudas GR, Wan Y, Davoudi A (2022) Anomaly detection and correction of optimizing autonomous systems with inverse reinforcement learning. IEEE Trans. Cybernetics 53(7):4555–4566.Google Scholar
• MacLeod BP, Parlane FG, Morrissey TD, Häse F, Roch LM, Dettelbach KE, Moreira R, et al. (2020) Self-driving laboratory for accelerated discovery of thin-film materials. Sci. Adv. 6(20):eaaz8867.Google Scholar
• Merchant A, Batzner S, Schoenholz SS, Aykol M, Cheon G, Cubuk ED (2023) Scaling deep learning for materials discovery. Nature 624:80–85.Google Scholar
• Modirshanechi A, Brea J, Gerstner W (2022) A taxonomy of surprise definitions. J. Math. Psych. 110:102712.Google Scholar
• Montechiesi L, Cocconcelli M, Rubini R (2016) Artificial immune system via Euclidean distance minimization for anomaly detection in bearings. Mechanical Systems Signal Processing 76:380–393.Google Scholar
• Moreno-Torres JG, Raeder T, Alaiz-Rodríguez R, Chawla NV, Herrera F (2012) A unifying view on dataset shift in classification. Pattern Recognition 45(1):521–530.Google Scholar
• Nguyen DT, Lou Z, Klar M, Brox T (2019) Anomaly detection with multiple-hypotheses predictions. Proc. 36th Internat. Conf. Machine Learn., Proceedings of Machine Learning Research, vol. 97 (PMLR, New York), 4800–4809.Google Scholar
• Nikolaev P, Hooper D, Webber F, Rao R, Decker K, Krein M, Poleski J, et al. (2016) Autonomy in materials research: A case study in carbon nanotube growth. NPJ Comput. Material 2:16031.Google Scholar
• Paninski L (2003) Estimation of entropy and mutual information. Neural Comput. 15(6):1191–1253.Google Scholar
• Park HS, Tran NH (2012) An autonomous manufacturing system based on swarm of cognitive agents. J. Manufacturing Systems 31(3):337–348.Google Scholar
• Prat-Carrabin A, Wilson RC, Cohen JD, Azeredo da Silveira R (2021) Human inference in changing environments with temporal structure. Psych. Rev. 128(5):879–912.Google Scholar
• Raihan AS, Khosravi H, Bhuiyan TH, Ahmed I (2024) An augmented surprise-guided sequential learning framework for predicting the melt pool geometry. J. Manufacturing Systems 75:56–77.Google Scholar
• Reis J, Cohen Y, Melão N, Costa J, Jorge D (2021) High-tech defense industries: Developing autonomous intelligent systems. Appl. Sci. 11(11):4920.Google Scholar
• Rousseeuw PJ, Croux C (1993) Alternatives to the median absolute deviation. J. Amer. Statist. Assoc. 88(424):1273–1283.Google Scholar
• Schlegl T, Seeböck P, Waldstein SM, Langs G, Schmidt-Erfurth U (2019) F-anoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical Image Anal. 54:30–44.Google Scholar
• Shannon CE (1948) A mathematical theory of communication. Bell System Tech. J. 27(3):379–423.Google Scholar
• Sugiyama M, Krauledat M, Müller KR (2007) Covariate shift adaptation by importance weighted cross validation. J. Machine Learn. Res. 8(5):985–1005.Google Scholar
• Szymanski NJ, Rendy B, Fei Y, Kumar RE, He T, Milsted D, McDermott MJ, et al. (2023) An autonomous laboratory for the accelerated synthesis of novel materials. Nature 624:86–91.Google Scholar
• Verdú S (1994) Generalizing the Fano inequality. IEEE Trans. Inform. Theory 40(4):1247–1251.Google Scholar
• Wang Y, Miao Q, Ma EW, Tsui KL, Pecht MG (2013) Online anomaly detection for hard disk drives based on Mahalanobis distance. IEEE Trans. Reliability 62(1):136–145.Google Scholar
• Weller-Fahy DJ, Borghetti BJ, Sodemann AA (2014) A survey of distance and similarity measures used within network intrusion anomaly detection. IEEE Comm. Surveys Tutorials 17(1):70–91.Google Scholar
• Yurtsever E, Lambert J, Carballo A, Takeda K (2020) A survey of autonomous driving: Common practices and emerging technologies. IEEE Access 8:58443–58469.Google Scholar
• Zamiri-Jafarian Y, Plataniotis KN (2022) A Bayesian surprise approach in designing cognitive radar for autonomous driving. Entropy 24(5):672.Google Scholar
• Zhang K, Bui AT, Apley DW (2023) Concept drift monitoring and diagnostics of supervised learning models via score vectors. Technometrics 65(2):137–149.Google Scholar
• Zhou ZG, Tang P (2016) Continuous anomaly detection in satellite image time series based on z-scores of season-trend model residuals. Proc. IEEE Internat. Geosci. Remote Sensing Sympos. (IEEE, Piscataway, NJ), 3410–3413.Google Scholar
• Žliobaitė I, Pechenizkiy M, Gama J (2016) An overview of concept drift applications. Big Data Analysis: New Algorithms New Society 16: 91–114.Google Scholar