结合深度学习的监督主题模型

doi:10.16451/j.cnki.issn1003-6059.201808004

摘要
图/表
参考文献(34)
相关文章 (10)

全文: PDF (830 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

无监督主题模型在降维过程中缺少标签信息的指导,丢失一些具有判别性的文本特征,导致最终的分类结果不理想.因此,文中提出结合深度学习的监督主题模型,利用深度网络强大的非线性拟合能力建立文档主题分布与标签之间的映射,利用变分期望最大化(EM)和深度网络训练方法共同完成贝叶斯框架下模型参数的更新,通过改变网络结构和激活函数的类型,用于分类和回归任务.实验表明文中模型既能保持无监督主题模型隐含主题的提取能力,还能更好地完成分类和回归任务.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	苑东东
	赵杰煜
	叶绪伦

关键词 ：监督主题模型, 深度学习, 变分期望最大化(EM)算法

Abstract：

Some discriminative text features are lost due to the lack of guidance of label information during dimensionality reduction in unsupervised topic models. Thus, the classification result is unsatisfactory. A supervised topic model with deep learning, namely supervised latent Dirichlet allocation with deep learning(DL-sLDA) is proposed in this paper. A deep neural network is exploited to establish the mapping between the topic of a document and its label. Both variational expectation-maximization and deep learning method are adopted to update the defined parameters under Bayesian framework in DL-sLDA. When the structure of the deep network and the type of the activation function are changed properly, the proposed model can be utilized for both classification and regression tasks. The experimental result demonstrates that DL-sLDA maintains the ability of topic extraction and gains a better predictive ability.

Key words： Supervised Topic Model Deep Learning Variational Expectation-Maximization(EM) Algorithm

收稿日期: 2018-01-17

ZTFLH:

TP 391

基金资助:

国家自然科学基金项目(No.61571247)、浙江省自然科学基金重点项目(No.LZ16F030001)、浙江省国际合作专项目(No.2013C24027)资助

通讯作者: 赵杰煜,博士,教授,主要研究方向为图形图像技术、自然交互、机器视觉.E-mail:zhao_jieyu@nbu.edu.cn.

作者简介: 苑东东,硕士研究生,主要研究方向为机器学习、自然语言处理.E-mail:496483534@qq.com. 叶绪伦,博士研究生,主要研究方向为流行学习、非负矩阵分解.E-mail:yexlwh@163.com.

引用本文:

苑东东, 赵杰煜, 叶绪伦. 结合深度学习的监督主题模型[J]. 模式识别与人工智能, 2018, 31(8): 715-724. YUAN Dongdong, ZHAO Jieyu, YE Xulun. Supervised Topic Model with Deep Learning. , 2018, 31(8): 715-724.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.201808004 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2018/V31/I8/715

[1] BLEI D M, MCAULIFFE J D. Supervised Topic Models[C/OL]. [2017-12-20]. https://arxiv.org/pdf/1003.0783.pdf.
[2] ZHU J, AHMED A, XING E P. MedLDA: Maximum Margin Supervised Topic Models. Journal of Machine Learning Research, 2012, 13: 2237-2278.
[3] RODRIGUES F, LOURENCO M, RIBEIRO B, et al. Learning Supervised Topic Models for Classification and Regression from Crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2409-2422.
[4] WANG C, BLEI D, LI F F. Simultaneous Image Classification and Annotation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2009: 1903-1910.
[5] HOU S, CHEN L, TAO D, et al. Multi-layer Multi-view Topic Model for Classifying Advertising Video. Pattern Recognition, 2017, 68: 66-81.
[6] CHEN C, ZARE A, TRINH H N, et al. Partial Membership Latent Dirichlet Allocation for Soft Image Segmentation. IEEE Transactions on Image Processing, 2017, 26(12): 5590-5602.
[7] WANG J D, ZHAO Z, ZHOU J Z, et al. Recommending Flickr Groups with Social Topic Model. Information Retrieval, 2012, 15(3/4): 278-295.
[8] QIU Z C, SHEN H. User Clustering in a Dynamic Social Network Topic Model for Short Text Streams. Information Sciences, 2017, 414: 102-116.
[9] BLEI D M, NG A Y, JORDAN M I. Latent Dirichlet Allocation. Journal of Machine Learning Research, 2003, 3: 993-1022.
[10] BLEI D M, LAFFERTY J D. Correlated Topic Models[C/OL]. [2017-12-20]. http://people.ee.duke.edu/~lcarin/Blei2005CTM.pdf.
[11] LI W, MCCALLUM A. Pachinko Allocation: DAG-Structured Mixture Models of Topic Correlations // Proc of the 23rd International Conference on Machine Learning. New York, USA: ACM, 2006: 577-584.
[12] BLEI D M, LAFFERTY J D. Dynamic Topic Models // Proc of the 23rd International Conference on Machine Learning. New York, USA: ACM, 2006: 113-120.
[13] WANG C, BLEI D, HECKERMAN D. Continuous Time Dynamic Topic Models // Proc of the 24th Conference on Uncertainty in Artificial Intelligence. Helsinki, Finland: AUAI, 2008: 579-586.
[14] ZHAO W X, JIANG J, WENG J S, et al. Comparing Twitter and Traditional Media Using Topic Models // Proc of the European Conference on Information Retrieval. Berlin, Germany: Springer, 2011: 338-349.
[15] LIU G L, XU X F, ZHU Y, et al. An Improved Latent Dirichlet Allocation Model for Hot Topic Extraction // Proc of the 4th IEEE International Conference on Big Data and Cloud Computing. Wa-shington, USA: IEEE, 2015: 470-476.
[16] CAO Z Q, LI S J, LIU Y, et al. A Novel Neural Topic Model and Its Supervised Extension // Proc of the 29th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2015: 2210-2216.
[17] LACOSTE-JULIEN S, SHA F, JORDAN M I. DiscLDA: Discri-minative Learning for Dimensionality Reduction and Classification // KALLER D, SCHUURMANS D, BENGIO Y, et al., eds. Advances in Neural Information Processing Systems 21. Cambridge, USA: The MIT Press, 2009: 897-904.
[18] TADDY M. Multinomial Inverse Regression for Text Analysis. Journal of the American Statistical Association, 2013, 108(503): 755-770.
[19] RABINOVICH M, BLEI D M. The Inverse Regression Topic Model // Proc of the 31st International Conference on Machine Learning. New York, USA: ACM, 2014: 199-207.
[20] ZHANG C, EK CARL H, GRATAL X, et al. Supervised Hierarchical Dirichlet Processes with Variational Inference // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2013: 254-261.
[21] DAI A M, STORKEY A J. The Supervised Hierarchical Dirichlet Process. IEEE Transactions on Pattern Analysis and Machine Inte-lligence, 2015, 37(2): 243-255.
[22] REN Y, WANG Y N, ZHU J. Spectral Learning for Supervised Topic Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3): 726-739.
[23] KANDEMIR M, KEKEÇ T, YENITERZI R. Supervising Topic Models with Gaussian Processes. Pattern Recognition, 2018, 77: 226-236.
[24] SEBER G A F. Nonlinear Regression Models // SEBER G A F, ed. The Linear Model and Hypothesis. Berlin, Germany: Sprin-ger, 2015: 117-128 .
[25] LECUN Y, BENGIO Y, HINTON G. Deep Learning. Nature, 2015, 521(7553): 436-444.
[26] FARABET C, COUPRIE C, NAJMAN L, et al. Learning Hierarchical Features for Scene Labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1915-1929.
[27] ER MENG J, ZHANG Y, WANG N, et al. Attention Pooling-Based Convolutional Neural Network for Sentence Modelling. Information Sciences, 2016, 373: 1339-1351.
[28] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778.
[29] BLEI D M, JORDAN M I. Modeling Annotated Data // Proc of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2003: 127-134.
[30] JOACHIMS T. Text Categorization with Support Vector Machines: Learning with Many Relevant Features // Proc of the European Conference on Machine Learning. Berlin, Germany: Springer, 1998: 137-142.
[31] LI X M, OUYANG J H, ZHOU X T, et al. Supervised Labeled Latent Dirichlet Allocation for Document Categorization. Applied Intelligence, 2015, 42(3): 581-593.
[32] LAI S W, XU L H, LIU K, et al. Recurrent Convolutional Neural Networks for Text Classification // Proc of the 29th AAAI National Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2015: 2267-2273.
[33] IYYER M, MANJUNATHA V, BOYD-GRABER J, et al. Deep Unordered Composition Rivals Syntactic Methods for Text Classification // Proc of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confe-rence on Natural Language Processing. Stroudsburg, USA: ACL, 2015: 1681-1691.
[34] PANG B, LEE L. Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales // Proc of the 43rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2005: 115-124.