Abstract To solve the problems of weak perception for aspect words and generalization ability in the existing models for sentiment classification, a context-oriented attention joint learning network for aspect-level sentiment classification(CAJLN) is proposed. Firstly, the bidirectional encoder representation from transformers(BERT) model is employed as the encoder to preprocess short texts into sentences, sentence pairs and aspect words as input, and their hidden features are extracted through the single sentence and sentence pair classification models, respectively. Then, based on the hidden features of sentences and aspect words, attention mechanisms for sentences and aspect words are established to obtain aspect-specific context-aware representation. Then, the hidden features of sentence pairs and aspect-specific context-aware representations are learned jointly. Xavier normal distribution is utilized to initialize the weights. Thus, the continuous updating of the parameters during the back propagation is ensured, and useful information is learned by CAJLN in the training process. Experiments show that CAJLN effectively improves the performance of sentiment classification for short text on multiple datasets.
Fund:Supported by National Natural Science Foundation of China(No.61876158)
Corresponding Authors:
FENG Lin, Ph.D., professor. His research interests include machine learning and data mining.
About author:: YANG Yuting, master student. Her research interests include machine learning and data mining. DAI Leichao, master student. His research interests include machine learning and data mining.
[1] 陈 龙,管子玉,何金红,等.情感分类研究进展.计算机研究与发展, 2017, 54(6): 1150-1170. (CHEN L, GUAN Z Y, HE J H, et al. A Survey on Sentiment Classification. Journal of Computer Research and Development, 2017, 54(6): 1150-1170.) [2] PANG B, LEE L. Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval, 2008, 2(1/2): 1-135. [3] KIRITCHENKO S, ZHU X, CHERRY C, et al. NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews // Proc of the 8th International Workshop on Semantic Evaluation. Berlin, Germany: Springer, 2014: 437-442. [4] ZHANG M S, ZHANG Y, VO D T. Gated Neural Networks for Targeted Sentiment Analysis // Proc of the 30th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2016: 3087-3093. [5] TANG D Y, QIN B, FENG X C, et al. Effective LSTMs for Target-Dependent Sentiment Classification // Proc of the 26th International Conference on Computational Linguistics(Technical Papers). Stroudsburg, USA: ACL, 2016: 3298-3307. [6] MA Y K, PENG H Y, CAMBRIA E. Targeted Aspect-Based Sentiment Analysis via Embedding Commonsense Knowledge into an Attentive LSTM // Proc of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2018: 5876-5883. [7] MNIH V, HEESS N, GRAVES A, et al. Recurrent Models of Visual Attention // Proc of the 28th Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 2204-2212. [8] MA D H, LI S J, ZHANG X D, et al. Interactive Attention Networks for Aspect-Level Sentiment Classification // Proc of the 26th International Joint Conference on Artificial Intelligence. New York, USA: ACM, 2017: 4068-4074. [9] 曾义夫,蓝 天,吴祖峰,等.基于双记忆注意力的方面级别情感分类模型.计算机学报, 2019, 42(8): 1845-1857. (ZENG Y F, LAN T, WU Z F, et al. Bi-memory Based Attention Model for Aspect Level Sentiment Classification. Chinese Journal of Computers, 2019, 42(8): 1845-1857.) [10] 武 婷,曹春萍.融合位置权重的基于注意力交叉注意力的长短期记忆方面情感分析模型.计算机应用, 2019, 39(8): 2198-2203. (WU T, CAO C P. Aspect Level Sentiment Classification Model with Location Weight and Long-Short Term Memory Based on Attention-over-Attention. Journal of Computer Applications. 2019, 39(8): 2198-2203.) [11] WANG Y Q, HUANG M L, ZHU X Y, et al. Attention-Based LSTM for Aspect-Level Sentiment Classification // Proc of the International Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2016: 606-615. [12] CHEN P, SUN Z Q, BING L P, et al. Recurrent Attention Network on Memory for Aspect Sentiment Analysis // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2017: 452-461. [13] 王行甫,王 磊,苗付友,等.结合词性、位置和单词情感的内存网络的方面情感分析.小型微型计算机系统, 2019, 40(2): 383-389. (WANG X F, WANG L, MIAO F Y, et al. Aspect Level Sentiment Analysis with Memory Network with POS, Position and Polarity of Word. Journal of Chinese Computer Systems, 2019, 40(2): 383-389.) [14] 杜 慧,俞晓明,刘 悦,等.融合词性和注意力的卷积神经网络对象级情感分类方法.模式识别与人工智能, 2018, 31(12): 1120-1126. (DU H, YU X M, LIU Y, et al. CNN with Part-of-Speech and Attention Mechanism for Targeted Sentiment Classification. Pattern Recognition and Artificial Intelligence, 2018, 31(12): 1120-1126.) [15] 张新生,高 腾.多头注意力记忆网络的对象级情感分类.模式识别与人工智能, 2019, 32(11): 997-1005. (ZHANG X S, GAO T. Aspect Level Sentiment Classification with Multiple-Head Attention Memory Network. Pattern Recognition and Artificial Intelligence, 2019, 32(11): 997-1005.) [16] MIKOLOV T, SUTSKEVER L, CHEN K, et al. Distributed Re-presentations of Words and Phrases and Their Compositionality // Proc of the 26th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2013: 3111-3119. [17] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All you Need // Proc of the 31th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2017: 5998-6008. [18] PETERS M, NEUMANN M, IYYER M, et al. Deep Contextua-lized Word Representations // Proc of the Conference on North American Chapter of the Association for Computational Linguistics(Human Language Technologies). Stroudsburg, USA: ACL, 2018: 2227-2237. [19] RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving Language Understanding by Generative Pre-Training[C/OL]. [2020- 04-26]. https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf. [20] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding // Proc of the Conference of the North American Chapter of the Association for Computational Linguistics(Human Language Technologies). Stroudsburg, USA: ACL, 2019: 4171-4186. [21] WERBOS P J. Backpropagation through Time: What It Does and How to Do It. Proceeding of the IEEE, 1990, 78(10): 1550-1560. [22] GLOROT X, BENGIO Y. Understanding the Difficulty of Training Deep Feedforward Neural Networks // Proc of the 13th Internatio-nal Conference on Artificial Intelligence and Statistics. Berlin, Germany: Springer, 2010: 249-256. [23] XU K, BA J L, KIROS R, et al. Show, Attention and Tell: Neural Image Caption Generation with Visual Attention // Proc of the 32nd International Conference on Machine Learning. Stroudsburg, USA: ACL, 2015: 2048-2057. [24] LUONG M, PHAM H, MANNING C D, et al. Effective Approaches to Attention-Based Neural Machine Translation // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2015: 1412-1421. [25] ZENG B Q, YANG H, XU R Y, et al. LCF: A Local Context Focus Mechanism for Aspect-Based Sentiment Classification. Applied Sciences, 2019, 9(16): 3389-3408. [26] DONG L, WEI F R, TAN C Q, et al. Adaptive Recursive Neural Network for Target-Dependent Twitter Sentiment Classification // Proc of the 52nd Annual Meeting of the Association for Computational Linguistics(Short Papers). Stroudsburg, USA: ACL, 2014, II: 49-54.
2020, Vol. 33 
