A Theory-Based Explainable Deep Learning Architecture for Music Emotion

References

• Aljanaki A, Yang YH, Soleymani M (2017) Developing a benchmark for emotional analysis of music. PLoS One 12(3):e0173392.Crossref, Google Scholar
• Allan D (2008) A content analysis of music placement in prime-time television advertising. J. Advertising Res. 48(3):404–417.Crossref, Google Scholar
• Andrade EB (2005) Behavioral consequences of affect: Combining evaluative and regulatory mechanisms. J. Consumer Res. 32(3):355–362.Crossref, Google Scholar
• Belanche D, Flavián C, Pérez-Rueda A (2017) Understanding interactive online advertising: Congruence and product involvement in highly and lowly arousing, skippable video ads. J. Interactive Marketing 37(1):75–88.Crossref, Google Scholar
• Blaszke M, Kostek B (2022) Musical instrument identification using deep learning approach. Sensors 22(8):3033.Crossref, Google Scholar
• Boughanmi K, Ansari A (2021) Dynamics of musical success: A machine learning approach for multimedia data fusion. J. Marketing Res. 58(6):1034–1057.Crossref, Google Scholar
• Brigato L, Iocchi L (2021) A close look at deep learning with small data. 2020 25th Internat. Conf. Pattern Recognition (ICPR) (IEEE, Piscataway, NJ), 2490–2497.Google Scholar
• Bruner GC (1990) Music, mood, and marketing. J. Marketing 54(4):94–104.Crossref, Google Scholar
• Bullerjahn C, Güldenring M (1994) An empirical investigation of effects of film music using qualitative content analysis. Psychomusicology 13(1–2):99–118.Crossref, Google Scholar
• Chakraborty I, Chiong K, Dover H, Sudhir K (2024) Can AI and AI-Hybrids detect persuasion skills? Salesforce hiring with conversational video interviews. Marketing Sci. Forthcoming.Link, Google Scholar
• Chen L, Lee CM (2017) Convolutional neural network for humor recognition. Preprint, submitted February 8, https://www.researchgate.net/publication/313519600_Convolutional_Neural_Network_for_Humor_Recognition.Google Scholar
• Choi K, Fazekas G, Sandler M, Cho K (2017) Convolutional recurrent neural networks for music classification. 2017 IEEE Internat. Conf. Acoustics Speech Signal Processing (ICASSP) (IEEE, Piscataway, NJ), 2392–2396.Google Scholar
• Choi K, Fazekas G, Cho K, Sandler M (2018) A tutorial on deep learning for music information retrieval. Preprint, submitted May 3, https://arxiv.org/abs/1709.04396.Google Scholar
• Chowdhury S, Vall A, Haunschmid V, Widmer G (2019) Toward explainable music emotion recognition: The route via mid-level features. Preprint, submitted July 8, https://arxiv.org/abs/1907.03572.Google Scholar
• Christensen T (2006) The Cambridge History of Western Music Theory (Cambridge University Press, Cambridge, UK).Google Scholar
• Cohen JB, Pham MT, Andrade EB (2018) The nature and role of affect in consumer behavior. Jansson-Boyd CV, Zawisza MJ, eds. Routledge International Handbook of Consumer Psychology (Routledge, London), 306–357.Google Scholar
• Corrigall KA, Schellenberg EG (2013) Music: The language of emotion. Handbook of Psychology of Emotions: Recent Theoretical Perspectives and Novel Empirical Findings, vol. 2 (Nova Hauppauge, New York), 299–325.Google Scholar
• Coulter KS (1998) The effects of affective responses to media context on advertising evaluations. J. Advertising 27(4):41–51.Crossref, Google Scholar
• Davani AM, Díaz M, Prabhakaran V (2022) Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguistics 10:92–110.Crossref, Google Scholar
• Dew R, Ansari A, Toubia O (2022) Letting logos speak: Leveraging multiview representation learning for data-driven branding and logo design. Marketing Sci. 41(2):401–425.Link, Google Scholar
• Eerola T, Vuoskoski JK (2011) A comparison of the discrete and dimensional models of emotion in music. Psych. Music 39(1):18–49.Crossref, Google Scholar
• Frade JLH, de Oliveira JHC, Giraldi JdME (2021) Advertising in streaming video: An integrative literature review and research agenda. Telecomm. Policy 45(9):102186.Crossref, Google Scholar
• Fu Z, Lu G, Ting KM, Zhang D (2010) A survey of audio-based music classification and annotation. IEEE Trans. Multimedia 13(2):303–319.Crossref, Google Scholar
• Gabrielsson A (2016) The relationship between musical structure and perceived expression. Hallam S, Cross I, Thaut M, eds. The Oxford Handbook of Music Psychology, 2nd ed. (Oxford University Press, Oxford, UK), 215–232.Google Scholar
• Gabrielsson A, Lindström E (2010) The role of structure in the musical expression of emotions. Juslin PN, ed. Handbook of Music and Emotion: Theory, Research, Applications (Oxford Academic, Oxford, UK), 367–400.Google Scholar
• Gerg ID, Monga V (2021) Structural prior driven regularized deep learning for sonar image classification. IEEE Trans. Geoscience Remote Sensing 60:1–16.Crossref, Google Scholar
• Gomez P, Danuser B (2007) Relationships between musical structure and psychophysiological measures of emotion. Emotion 7(2):377–387.Crossref, Google Scholar
• Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep Learning, vol. 1 (MIT Press, Cambridge, MA).Google Scholar
• Goodrich K, Schiller SZ, Galletta D (2015) Consumer reactions to intrusiveness of online-video advertisements: Do length, informativeness, and humor help (or hinder) marketing outcomes? J. Advertising Res. 55(1):37–50.Crossref, Google Scholar
• Gorn GJ (1982) The effects of music in advertising on choice behavior: A classical conditioning approach. J. Marketing 46(1):94–101.Crossref, Google Scholar
• Herget AK (2021) On music’s potential to convey meaning in film: A systematic review of empirical evidence. Psych. Music 49(1):21–49.Crossref, Google Scholar
• Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-r, Jaitly N, Senior A, et al. (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine 29(6):82–97.Crossref, Google Scholar
• Holbrook MB, Batra R (1987) Assessing the role of emotions as mediators of consumer responses to advertising. J. Consumer Res. 14(3):404–420.Crossref, Google Scholar
• Huang JT, Kaul R, Narayanan S (2022) Variety and risk-taking in content creation: Evidence from a field experiment using image recognition techniques. Preprint, submitted June 9, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4126834.Google Scholar
• Huron D (1989) Music in advertising: An analytic paradigm. Music Quart. 73(4):557–574.Crossref, Google Scholar
• Jaquet L, Danuser B, Gomez P (2014) Music and felt emotions: How systematic pitch level variations affect the experience of pleasantness and arousal. Psych. Music 42(1):51–70.Crossref, Google Scholar
• Johnson-Laird PN, Oatley K (2016) Emotions in music, literature, and film. Feldman Barrett L, Lewis M, Haviland-Jones JM, eds. Handbook of Emotions, 4th ed. (Guilford Press, New York), 82–97.Google Scholar
• Kallinen K, Ravaja N (2006) Emotion perceived and emotion felt: Same and different. Music Sci. 10(2):191–213.Crossref, Google Scholar
• Kamins MA, Marks LJ, Skinner D (1991) Television commercial evaluation in the context of program induced mood: Congruency vs. consistency effects. J. Advertising 20(2):1–14.Crossref, Google Scholar
• Kapoor A, Narayanan S, Sharma A (2022) Does emotional matching between video ads and content lead to better engagement: Evidence from a large-scale field experiment. Working paper, Stanford University, Palo Alto, CA.Google Scholar
• Kim YE, Schmidt EM, Migneco R, Morton BG, Richardson P, Scott J, Speck JA, Turnbull D (2010) Music emotion recognition: A state of the art review. 11th Internat. Soc. Music Inform. Retrieval Conf. (ISMIR 2010), vol. 86, 937–952.Google Scholar
• Krishna A (2012) An integrative review of sensory marketing: Engaging the senses to affect perception, judgment and behavior. J. Consumer Psych. 22(3):332–351.Crossref, Google Scholar
• Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Adv. Neural Inform. Processing Systems 25 (Lake Tahoe, NV), 1097–1105.Google Scholar
• Kutz JN, Brunton SL (2022) Parsimony as the ultimate regularizer for physics-informed machine learning. Nonlinear Dynam. 107(3):1801–1817.Crossref, Google Scholar
• Lee CJ, Andrade EB, Palmer SE (2013) Interpersonal relationships and preferences for mood-congruency in aesthetic experiences. J. Consumer Res. 40(2):382–391.Crossref, Google Scholar
• Li H, Lo HY (2015) Do you recognize its brand? The effectiveness of online in-stream video advertisements. J. Advertising 44(3):208–218.Crossref, Google Scholar
• Liu L, Dzyabura D, Mizik N (2020) Visual listening in: Extracting brand image portrayed on social media. Marketing Sci. 39(4):669–686.Link, Google Scholar
• Liu X, Chen Q, Wu X, Liu Y, Liu Y (2017) CNN based music emotion classification. Preprint, submitted April 19, https://arxiv.org/pdf/1704.05665.Google Scholar
• McAdams S, Giordano BL (2015) The perception of musical timbre. Hallam S, Cross I, Thaut M, eds. The Oxford Handbook of Music Psychology, 2nd ed. (Oxford University Press, Oxford, UK), 113–124.Google Scholar
• Melzner J, Raghubir P (2023) The sound of music: The effect of timbral sound quality in audio logos on brand personality perception. J. Marketing Res. 60(5):932–949.Crossref, Google Scholar
• Müller M (2015) Fundamentals of Music Processing: Audio, Analysis, Algorithms, Applications (Springer, Cham, Switzerland).Crossref, Google Scholar
• Nelson DJ, Grazier R, Paglia J, Perkowitz S (2013) Hollywood Chemistry: When Science Met Entertainment (American Chemical Society, Washington, DC).Crossref, Google Scholar
• Panda R, Malheiro R, Paiva RP (2018) Novel audio features for music emotion recognition. IEEE Trans. Affective Comput. 11(4):614–626.Crossref, Google Scholar
• Pavan MC, Dos Santos VG, Lan AG, Martins J, Santos WR, Deutsch C, Costa PB, Hsieh FC, Paraboni I (2023) Morality classification in natural language text. IEEE Trans. Affective Comput. 14(2):857–863.Crossref, Google Scholar
• Pons J, Lidy T, Serra X (2016) Experimenting with musically motivated convolutional neural networks. 2016 14th Internat. Workshop Content-Based Multimedia Indexing (CBMI) (IEEE), 1–6.Google Scholar
• Puccinelli NM, Wilcox K, Grewal D (2015) Consumers’ response to commercials: When the energy level in the commercial conflicts with the media context. J. Marketing 79(2):1–18.Crossref, Google Scholar
• Rajaram P, Manchanda P (2024) Unboxing engagement in YouTube influencer videos: An attention-based approach. Preprint, submitted August 6, https://arxiv.org/pdf/2012.12311.Google Scholar
• Ribeiro MT, Singh S, Guestrin C (2016) “Why should I trust you?” Explaining the predictions of any classifier. Proc. 22nd ACM SIGKDD Internat. Conf. (Association for Computing Machinery, New York), 1135–1144.Google Scholar
• Russell JA (1980) A circumplex model of affect. J. Personality Soc. Psych. 39(6):1161–1178.Crossref, Google Scholar
• Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2020) Grad-CAM: Visual explanations from deep networks via gradient-based localization. Internat. J. Comput. Vision 128:336–359.Google Scholar
• Sisodia A, Burnap A, Kumar V (2024) Automatic discovery and generation of visual design characteristics: Application to visual conjoint. Preprint, submitted August 3, http://dx.doi.org/10.2139/ssrn.4151019.Google Scholar
• Stoppe S (2014) Film in Concert. Film Scores and Their Relation to Classical Concert Music (Verlag Werner Hülsbusch, Hamburg, Germany).Google Scholar
• Teeny JD, Siev JJ, Briñol P, Petty RE (2021) A review and conceptual framework for understanding personalized matching effects in persuasion. J. Consumer Psych. 31(2):382–414.Crossref, Google Scholar
• Thompson WF, Balkwill LL (2010) Cross-cultural similarities and differences. Juslin PN, Sloboda JA, eds. Handbook of Music and Emotion: Theory, Research, Applications (Oxford University Press, Oxford, UK), 755–788.Google Scholar
• Toubia O, Berger J, Eliashberg J (2021) How quantifying the shape of stories predicts their success. Proc. Natl. Acad. Sci. USA 118(26):e2011695118.Crossref, Google Scholar
• Troncoso I, Luo L (2023) Look the part? The role of profile pictures in online labor markets. Marketing Sci. 42(6):1080–1100.Link, Google Scholar
• Unni D, D’Cunha AM, Deepa G (2022) A technique to detect music emotions based on machine learning classifiers. 2022 Second Internat. Conf. Interdisciplinary Cyber Physical Systems (ICPS) (IEEE), 136–140.Google Scholar
• Wang X, He J, Curry DJ, Ryoo JH (2022) Attribute embedding: Learning hierarchical representations of product attributes from consumer reviews. J. Marketing 86(6):155–175.Crossref, Google Scholar
• Wilbur KC (2008) A two-sided, empirical model of television advertising and viewing markets. Marketing Sci. 27(3):356–378.Link, Google Scholar
• Yang Y, Chen H (2011) Predicting the distribution of perceived emotions of a music signal for content retrieval. IEEE Trans. Audio Speech Language Processing 19(7):2184–2196.Crossref, Google Scholar
• Yang J, Zhang J, Zhang Y (2023) First law of motion: Influencer video advertising on TikTok. Preprint, submitted August 18, http://dx.doi.org/10.2139/ssrn.3815124.Google Scholar
• Yang J, Xie Y, Krishnamurthi L, Papatla P (2022) High-energy ad content: A large-scale investigation of TV commercials. J. Marketing Res. 59(4):840–859.Crossref, Google Scholar
• Zhang M, Luo L (2023) Can consumer-posted photos serve as a leading indicator of restaurant survival? Evidence from Yelp. Management Sci. 69(1):25–50.Link, Google Scholar