HeBERT and HebEMO: A Hebrew BERT Model and a Tool for Polarity Analysis and Emotion Recognition

References

• Abdul-Mageed M, Diab M, Korayem M (2011) Subjectivity and sentiment analysis of modern standard Arabic. Proc. 49th Annual Meeting Assoc. Comput. Linguistics: Human Language Tech. (Association for Computational Linguistics, Stroudsburg, PA), 587–591.Google Scholar
• Acheampong FA, Wenyu C, Nunoo-Mensah H (2020) Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports 2(7):e12189.Google Scholar
• Adamopoulos P, Ghose A, Todri V (2018) The impact of user personality traits on word of mouth: Text-mining social media platforms. Inform. Systems Res. 29(3):612–640.Link, Google Scholar
• Ahmad Z, Jindal R, Ekbal A, Bhattachharyya P (2020) Borrow from rich cousin: transfer learning for emotion detection using cross lingual embedding. Expert Systems Appl. 139:112851.Google Scholar
• Ahorsu DK, Lin CY, Imani V, Saffari M, Griffiths MD, Pakpour AH (2020) The fear of COVID-19 scale: development and initial validation. Internat. J. Mental Health Addiction, ePub ahead of print March 27.Google Scholar
• Amram A, David AB, Tsarfaty R (2018) Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from modern Hebrew. Proc. 27th Internat. Conf. Comput. Linguistics (Association for Computational Linguistics, Stroudsburg, PA), 2242–2252.Google Scholar
• Antoun W, Baly F, Hajj H (2020) AraBERT: Transformer-based model for Arabic language understanding. Proc. 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (European Language Resource Association, Marseille, France), 9–15.Google Scholar
• Argaman O (2010) Linguistic markers and emotional intensity. J. Psycholinguistic Res. 39(2):89–99.Google Scholar
• Bareket D, Tsarfaty R (2021) Neural modeling for named entities and morphology (NEMO2). Transactions of the Association for Computational Linguistics (MIT Press, Cambridge, MA), 9:909–928.Google Scholar
• Belinkov Y, Durrani N, Dalvi F, Sajjad H, Glass J (2017) What do neural machine translation models learn about morphology? Preprint, submitted April 11, https://arxiv.org/abs/1704.03471.Google Scholar
• Bellstam G, Bhagat S, Cookson JA (2020) A text-based analysis of corporate innovation. Management Sci. 67(7):4004–4031.Google Scholar
• Bojanowski P, Joulin A, Mikolov T (2015) Alternative structures for character-level RNNs. Preprint, submitted November 19, https://arxiv.org/abs/1511.06303.Google Scholar
• Chatterjee A, Narahari KN, Joshi M, Agrawal P (2019) SemEval-2019 task 3: EmoContext contextual emotion detection in text. Proc. 13th Internat. Workshop Semantic Eval. (Association for Computational Linguistics, Stroudsburg, PA), 39–48.Google Scholar
• Chitturi R, Raghunathan R, Mahajan V (2007) Form vs. function: How the intensities of specific emotions evoked in functional vs. hedonic trade-offs mediate product preferences. J. Marketing Res. 44(4):702–714. Google Scholar
• Chriqui, A, Yahav I (2021). HeBERT & HebEMO: A Hebrew BERT model and a tool for polarity analysis and emotion recognition. Preprint, submitted February 3, https://arxiv.org/abs/2102.01909.Google Scholar
• Desmet B, Hoste V (2013) Emotion detection in suicide notes. Expert Systems Appl. 40(16):6351–6358.Google Scholar
• Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: Pre-training of deep bidirectional transformers for language understanding. Preprint, submitted October 11, https://arxiv.org/abs/1810.04805.Google Scholar
• Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent Twitter sentiment classification. Proc. 52nd Annual Meeting Assoc. Comput. Linguistics, vol. 2 (Association for Computational Linguistics, Stroudsburg, PA), 49–54.Google Scholar
• Ekman P (1999) Basic emotions. Dalgleish T, Power MJ, eds. Handbook of Cognition and Emotion (Wiley, Hoboken, NJ), 45–60.Google Scholar
• El-Din DM (2016) Enhancement bag-of-words model for solving the challenges of sentiment analysis. J. Adv. Comput. Sci. Appl. 7(1): 244–252.Google Scholar
• Fattah K, Fierke KM (2009) A clash of emotions: The politics of humiliation and political violence in the middle east. Eur. J. Internat. Relations 15(1):67–93.Google Scholar
• Fedus W, Goodfellow I, Dai AM (2018) MaskGAN: Better text generation via filling in the _. Preprint, submitted January 23, https://arxiv.org/abs/1801.07736.Google Scholar
• Ghanbari-Adivi F, Mosleh M (2019) Text emotion detection in social networks using a novel ensemble classifier based on Parzen tree estimator (TPE). Neural Comput. Appl. 31(12):8971–8983.Google Scholar
• Hemmatian F, Sohrabi MK (2019) A survey on classification techniques for opinion mining and sentiment analysis. Artificial Intelligence Rev. 52:1495–1545.Google Scholar
• Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput. 9(8):1735–1780.Google Scholar
• Hogenboom A, Bal D, Frasincar F, Bal M, de Jong F, Kaymak U (2013) Exploiting emoticons in sentiment analysis. Proc. 28th Annual ACM Sympos. Appl. Comput. (Association for Computing Machinery, New York), 703–710.Google Scholar
• Jawahar G, Sagot B, Seddah D (2019) What does BERT learn about the structure of language? Proc. 57th Annual Meeting Assoc. Comput. Linguistics (Association for Computational Linguistics, Stroudsburg, PA), 3651–3657.Google Scholar
• Joulin A, Grave E, Bojanowski P, Douze M, Jégou H, Mikolov T (2016) FastText.zip: Compressing text classification models. Preprint, submitted December 12, https://arxiv.org/abs/1612.03651.Google Scholar
• Khan FH, Qamar U, Bashir S (2016) eSAP: A decision support framework for enhanced sentiment analysis and polarity classification. Inform. Sci. 367:862–873.Google Scholar
• Kim Y (2014) Convolutional neural networks for sentence classification. Preprint, submitted August 25, https://arxiv.org/abs/1408.5882.Google Scholar
• Kim-Prieto C, Diener E (2009) Religion as a source of variation in the experience of positive and negative emotions. J. Posit. Psychol. 4(6):447–460.Google Scholar
• Klein S, Tsarfaty R (2020) Getting the## life out of living: How adequate are word-pieces for modelling complex morphology? Proc. 17th SIGMORPHON Workshop Comput. Res. Phonetics, Phonology, Morphology (Association for Computational Linguistics, Stroudsburg, PA), 204–209.Google Scholar
• Kövecses Z (2003) Metaphor and Emotion: Language, Culture, and Body in Human Feeling (Cambridge University Press, Cambridge, MA).Google Scholar
• Kratzwald B, Ilić S, Kraus M, Feuerriegel S, Prendinger H (2018) Deep learning for affective computing: Text-based emotion recognition in decision support. Decision Support Systems 115:24–35.Google Scholar
• Krippendorff K (1970) Estimating the reliability, systematic error and random error of interval data. Ed. Psych. Measurement 30(1):61–70.Google Scholar
• Levine Y, Lenz B, Lieber O, Abend O, Leyton-Brown K, Tennenholtz M, Shoham Y (2020) PMI-masking: Principled masking of correlated spans. Preprint, submitted October 5, https://arxiv.org/abs/2010.01825.Google Scholar
• Li S, Ju S, Zhou G, Lin X (2012) Active learning for imbalanced sentiment classification. Proc. 2012 Joint Conf. Empirical Methods Natural Language Processing Comput. Natural Language Learn. (Association for Computational Linguistics, Stroudsburg, PA), 139–148.Google Scholar
• Liu B (2012) Sentiment Analysis and Opinion Mining. Synthesis Lectures on Human Language Technologies, vol. 5 (Morgan and Claypool, San Rafael, CA).Google Scholar
• Liu B, Zhang L (2012) A survey of opinion mining and sentiment analysis. Aggarwal C, Zhai C, eds. Mining Text Data (Springer, Boston), 415–463.Google Scholar
• Liu B, Blasch E, Chen Y, Shen D, Chen G (2013) Scalable sentiment classification for big data analysis using naive Bayes classifier. Proc. IEEE Internat. Conf. Big Data (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 99–104.Google Scholar
• Liu R, Shi Y, Ji C, Jia M (2019a) A survey of sentiment analysis based on transfer learning. IEEE Access 7:85401–85412.Google Scholar
• Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019b) RoBERTa: A robustly optimized BERT pretraining approach. Preprint, submitted July 26, https://arxiv.org/abs/1907.11692.Google Scholar
• Luo F, Li C, Cao Z (2016) Affective-feature-based sentiment analysis using SVM classifier. Proc. IEEE 20th Internat. Conf. Comput. Supported Cooperative Work Design (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 276–281.Google Scholar
• Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: A survey. Ain Shams Engrg. J. 5(4):1093–1113.Google Scholar
• Meyer C, Schwager A (2007) Understanding Customer Experience (Harvard Business Publishing, Boston).Google Scholar
• Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. Preprint, submitted January 16, https://arxiv.org/abs/1301.3781.Google Scholar
• Mohammad S, Bravo-Marquez F, Salameh M, Kiritchenko S (2018) SemEval-2018 task 1: Affect in tweets. Proc. 12th Internat. Workshop Semantic Eval. (Association for Computational Linguistics, Stroudsburg, PA), 1–17.Google Scholar
• Mohammad SM, Turney PD (2013) Crowdsourcing a word-emotion association lexicon. Comput. Intelligence 29(3):436–465.Google Scholar
• Mordecai NB, Elhadad M (2005) Hebrew named entity recognition. Preprint, submitted September, https://www.cs.bgu.ac.il/ ˜ elhadad/nlpproj/naama/HebNER.pdf.Google Scholar
• More A, Seker A, Basmova V, Tsarfaty R (2019) Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from modern Hebrew. Lee L, Johnson M, Roark B, Nenkova A, eds. Transactions of the Association for Computational Linguistics, vol. 7 (MIT Press, Cambridge, MA), 33–48.Google Scholar
• Mudinas A, Zhang D, Levene M (2012) Combining lexicon and learning based approaches for concept-level sentiment analysis. Proc. First Internat. Workshop Issues Sentiment Discovery Opinion Mining (Association for Computing Machinery, New York), 1–8.Google Scholar
• Mughaz D, Fuchs T, Bouhnik D (2018) Automatic opinion extraction from short Hebrew texts using machine learning techniques. Computación Sistemas 22(4):1347–1357.Google Scholar
• Ortiz Suárez PJ, Romary L, Sagot B (2020) A monolingual approach to contextualized word embeddings for mid-resource languages. Proc. 58th Annual Meeting Assoc. Comput. Linguistics (Association for Computational Linguistics, Stroudsburg, PA), 1703–1714.Google Scholar
• Ortony A, Clore GL, Foss MA (1987) The referential structure of the affective lexicon. Cognitive Sci. 11(3):341–364.Google Scholar
• Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans. Knowledge Data Engrg. 22(10):1345–1359.Google Scholar
• Patwa P, Aguilar G, Kar S, Pandey S, Pykl S, Gambäck B, Chakraborty T, Solorio T, Das A (2020) SemEval-2020 task 9: Overview of sentiment analysis of code-mixed tweets. Proc. Fourteenth Workshop on Semantic Evaluation, 774–790.Google Scholar
• Pedrosa AL, Bitencourt L, Fróes ACF, Cazumbá MLB, Campos RGB, de Brito SBCS, Simões E Silva AC (2020) Emotional, behavioral, and psychological impact of the COVID-19 pandemic. Frontiers Psych. 11:566212.Google Scholar
• Pennebaker JW, Francis ME, Booth RJ (2001) Linguistic Inquiry and Word Count: LIWC2001 (Lawrence Erlbaum Associates, Mahwah, NJ).Google Scholar
• Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. Preprint, submitted February 15, https://arxiv.org/abs/1802.05365.Google Scholar
• Pfefferbaum B, North CS (2020) Mental health and the COVID-19 pandemic. New England J. Medicine 383(6):510–512.Google Scholar
• Plutchik R (1980) A general psychoevolutionary theory of emotion. Plutchik R, Kellerman H, eds. Theories of Emotion (Elsevier, Amsterdam), 3–33.Google Scholar
• Pota M, Marulli F, Esposito M, De Pietro G, Fujita H (2019) Multilingual POS tagging by a composite deep architecture based on character-level features and on-the-fly enriched word embeddings. Knowledge-Based Systems 164:309–323.Google Scholar
• Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. OpenAI Blog (Jule 11), https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.Google Scholar
• Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI Blog (February 14), https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.Google Scholar
• Ren Y, Wang R, Ji D (2016) A topic-enhanced word embedding for Twitter sentiment classification. Inform. Sci. 369:188–198.Google Scholar
• Rosaldo MZ, Shweder RA, LeVine RA (1984) Culture Theory: Essays on Mind, Self, and Emotion (Cambridge University Press, Cambridge, MA).Google Scholar
• Schuster M, Nakajima K (2012) Japanese and Korean voice search. Proc. IEEE Internat. Conf. Acoustics, Speech Signal Processing (Institute of Electrical and Electronics Engineers, Piscataway, NJ), 5149–5152.Google Scholar
• Seker A, Bandel E, Bareket D, Brusilovsky I, Greenfeld RS, Tsarfaty R (2021) AlephBERT: A Hebrew large pre-trained language model to start-off your Hebrew NLP application with. Preprint, submitted April 8, https://arxiv.org/abs/2104.04052.Google Scholar
• Shapira N, Lazarus G, Goldberg Y, Gilboa-Schechtman E, Tuval-Mashiach R, Juravski D, Atzil-Slonim D (2020) Using computerized text analysis to examine associations between linguistic features and clients’ distress during psychotherapy. J. Counseling Psych. 68(1):77–87.Google Scholar
• Sima’an K, Itai A, Winter Y, Altman A, Nativ N (2001) Building a tree-bank of modern Hebrew text. Traitement automatique des langues 42(2):247–380.Google Scholar
• Steiner T (2016) President Rivlin’s “four tribes” initiative: The foreign policy implications of a democratic & inclusive process to address Israel’s socio-demographic transformation. Report, Institute for Policy and Strategy, Reichman University, Herzliya, Israel.Google Scholar
• Straka M, Hajic J, Straková J (2016) UDPipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. Proc. 10th Internat. Conf. Language Resources Evaluation (European Language Resources Association, Paris), 4290–4297.Google Scholar
• Tang D, Wei F, Yang N, Zhou M, Liu T, Qin B (2014) Learning sentiment-specific word embedding for Twitter sentiment classification. Proc. 52nd Annual Meeting Assoc. Comput. Linguistics, vol. 1 (Association for Computational Linguistics, Stroudsburg, PA), 1555–1565.Google Scholar
• Tay Y, Dehghani M, Bahri D, Metzler D (2020) Efficient transformers: A survey. Preprint, submitted September 14, https://arxiv.org/abs/2009.06732.Google Scholar
• Tripathy A, Agrawal A, Rath SK (2016) Classification of sentiment reviews using n-gram machine learning approach. Expert Systems Appl. 57:117–126.Google Scholar
• Tsarfaty R, Bareket D, Klein S, Seker A (2020) From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? Proc. 58th Annual Meeting of the Assoc. Comput. Linguistics (Association for Computational Linguistics - Online), 7396–7408.Google Scholar
• Tsarfaty R, Seddah D, Goldberg Y, Kübler S, Versley Y, Candito M, Foster J, Rehbein I, Tounsi L (2010) Statistical parsing of morphologically rich languages (SPMRL) what, how and whither. Proc. NAACL HLT 2010 First Workshop Statist. Parsing Morphologically-Rich Languages (Association for Computational Linguistics, Stroudsburg, PA), 1–12.Google Scholar
• Ullah R, Amblee N, Kim W, Lee H (2016) From valence to emotions: Exploring the distribution of emotions in online product reviews. Decision Support Systems 81:41–53.Google Scholar
• Vania C, Grivas A, Lopez A (2018) What do character-level models learn about morphology? The case of dependency parsing. Proc. 2018 Conf. Empirical Methods in Natural Language Processing (Association for Computational Linguistics, Brussels), 2573–2583.Google Scholar
• Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Guyon I, Von Luxburg U, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R, eds. Advances in Neural Information Processing Systems, vol. 30 (Curran Associates, Red Hook, NY), 5998–6008.Google Scholar
• Wang G, Zhang Z, Sun J, Yang S, Larson CA (2015) POS-RS: A random subspace method for sentiment classification based on part-of-speech analysis. Inform. Processing Management 51(4):458–479.Google Scholar
• Wierzbicka A (1994) Emotion, language, and cultural scripts. Kitayama S, Markus HR, eds. Emotion and Culture: Empirical Studies of Mutual Influence (American Psychological Association, Washington, DC), 133–196).Google Scholar
• Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, et al. (2020) Transformers: State-of-the-art natural language processing. Proc. 2020 Conf. Empirical Methods Natural Language Processing: System Demonstrations (Association for Computational Linguistics, Stroudsburg, PA), 38–45.Google Scholar
• Woodruff RB (1997) Customer value: The next source for competitive advantage. J. Acad. Marketing Sci. 25(2):139–153.Google Scholar
• Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, et al. (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. Preprint, submitted September 26, https://arxiv.org/abs/1609.08144.Google Scholar
• Yadav A, Vishwakarma DK (2020) Sentiment analysis using deep learning architectures: A review. Artificial Intelligence Rev. 53(6):4335–4385.Google Scholar
• Yamamoto Y, Kumamoto T, Nadamoto A (2014) Role of emoticons for multidimensional sentiment analysis of Twitter. Indrawan-Santiago M, Steinbauer M, Nguyen HQ, Tjoa AM, Khalil I, Anderst-Kotsis G, eds. Proc. 16th Internat. Conf. Inform. Integration Web-Based Appl. Services (Association for Computing Machinery, New York), 107–115.Google Scholar
• Yin W, Kann K, Yu M, Schütze H (2017) Comparative study of CNN and RNN for natural language processing. Preprint, submitted February 7, https://arxiv.org/abs/1702.01923.Google Scholar
• Yue L, Chen W, Li X, Zuo W, Yin M (2019) A survey of sentiment analysis in social media. Knowledge Inform. Systems 60:617–663.Google Scholar
• Zampieri M, Malmasi S, Nakov P, Rosenthal S, Farra N, Kumar R (2019) SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). Proc. 13th International Workshop on Semantic Evaluation (Association for Computational Linguistics, Minneapolis), 75–86.Google Scholar
• Zhang L, Wang S, Liu B (2018) Deep learning for sentiment analysis: A survey. WIREs Data Mining Knowledge Discovery 8(4):e1253.Google Scholar
• Zhong P, Wang D, Miao C (2019) Knowledge-enriched transformer for emotion detection in textual conversations. Proc. 2019 Conf. Empirical Methods in Natural Language Processing and the 9th Internat. Joint Conf. Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics, Hong Kong), 165–176.Google Scholar
• Zhou D, Wu S, Wang Q, Xie J, Tu Z, Li M (2020) Emotion classification by jointly learning to lexiconize and classify. Proc. 28th Internat. Conf. Comput. Linguistics (International Committee on Computational Linguistics, Barcelona, Spain), 3235–3245.Google Scholar