结合自注意力特征过滤分类器和双分支GAN的面部表情识别

doi:10.16451/j.cnki.issn1003-6059.202203004

摘要
图/表
参考文献
相关文章 (2)

全文: PDF (1153 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要现有面部表情识别方法提取表情特征时通常容易与其它面部属性混合,不利于面部表情的识别.对此,文中提出结合自注意力特征过滤分类器和双分支生成对抗网络的面部表情识别方法.首先,使用双分支生成对抗网络学习辨别性的表情表示,提出自注意力特征过滤分类器作为表情的分类模块.使用级联的LayerNorm和ReLU将低激活单元归零并保留高激活单元,生成多级特征.使用自注意力融合输出多级特征的预测结果,在一定程度上消除噪声对识别结果的影响.然后,提出基于滑动模块的双重图像一致性损失监督模型,学习具有辨别性的表情表示,使用滑动窗口计算重构损失,关注细节信息.最后,在CK+、RAF-DB、TFEID、BAUM-2i数据集上的实验表明文中方法识别效果较优.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	程艳
	蔡壮
	吴刚
	罗品
	邹海锋

关键词 ：面部表情识别, 双分支生成对抗网络, 自注意力特征过滤分类器, 滑动模块

Abstract：The expression features extracted by the existing facial expression recognition methods are usually mixed with other facial attributes, which is not conducive to facial expression recognition. A facial expression recognition model combining self-attention feature filter classifier and two-branch generative adversarial network is proposed. Two-branch generative adversarial network is introduced to learn discriminative expression representation, and a self-attention feature filtering classifier is proposed as the expression classification module. The cascaded LayerNorm and ReLU are employed to zero the low activation unit and retain the high activation unit to generate multi-level features. The self-attention is utilized to fuse and output the prediction results of multi-level features, and consequently the influence of noise on the recognition results is eliminated to a certain extent. A sliding module based dual image consistency loss supervised model is proposed to learn discriminative expression representations. The reconstruction loss is calculated by a sliding window and more attention is paid to the details. Finally, experiments on CK+, RAF-DB, TFEID and BAUM-2i datasets show the proposed model achieves better recognition results.

Key words： Facial Expression Recognition Two-Branch Generative Adversarial Network Self-Attention Feature Filtering Classifier Sliding Module

收稿日期: 2021-10-26

ZTFLH:

TP 391.41

基金资助:国家自然科学基金项目(No.62167006,61967011)、国家社会科学基金重点项目(No.20AXW009)、江西省自然科学基金项目(No.20202BABL202033,20212BAB202017)、江西省教育厅人文社科重点项目(No.JD19056)、江西省科技创新基地计划省重点实验室项目(No.20212BCD42001)、江西省03专项及5G项目(No.20212ABC03A22)、江西省主要学科学术和技术带头人培养计划领军人才项目(No.20213BCJL22047)资助

通讯作者: 程艳,博士,教授,主要研究方向为人工智能、智能信息处理、情感分析等.E-mail: chyan88888@jxnu.edu.cn.

作者简介: 蔡壮,硕士研究生,主要研究方向为面部表情识别、深度学习.E-mail:zhuangcai2020@163.com.
吴刚,硕士研究生,主要研究方向为深度学习、认知诊断.E-mail:1670760261@qq.com.
罗品,硕士研究生,主要研究方向为深度学习、情感分析.E-mail:497260662@qq.com.
邹海锋,硕士研究生,主要研究方向为深度学习、情感分析.E-mail:1766157791@qq.com.

引用本文:

程艳, 蔡壮, 吴刚, 罗品, 邹海锋. 结合自注意力特征过滤分类器和双分支GAN的面部表情识别[J]. 模式识别与人工智能, 2022, 35(3): 243-253. CHENG Yan, CAI Zhuang, WU Gang, LUO Pin, ZOU Haifeng. Facial Expression Recognition Combining Self-Attention Feature Filtering Classifier and Two-Branch GAN. Pattern Recognition and Artificial Intelligence, 2022, 35(3): 243-253.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202203004 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2022/V35/I3/243

[1] MEHRABIAN A.Communication without Words // MORTENSEN C D, ed. Communication Theory. 2nd Edition. New York,USA: Routledge, 2008: 193-200.
[2] LUCEY P, COHN J F, MATTHEWS L, et al. Automatically Detecting Pain in Video through Facial Action Units. IEEE Transactions on Systems, Man,and Cybernetics(Cybernetics), 2011, 41(3): 664-674.
[3] 龚礼林,刘红霞,赵蔚,等. 情感导学系统(ATS)的关键技术及其导学模型研究——论智能导学系统走向情感导学系统之意蕴.远程教育杂志, 2019, 37(5): 45-55.
(GONG L L, LIU H X, ZHAO W, et al. Research on Key Techniques of Affective Tutor System and Its Tutoring Model: The Implications from Intelligent Tutoring System to Affective Tutoring System. Journal of Distance Education, 2019, 37(5): 45-55.)
[4] SCHRODER M, BEVACQUA E, COWIE R, et al. Building Auto-nomous Sensitive Artificial Listeners. IEEE Transactions on Affective Computing, 2012, 3(2): 165-183.
[5] JEONG M, KO B C.Driver's Facial Expression Recognition in Real-Time for Safe Driving. Sensors, 2018, 18(12): 4270-4287.
[6] 王浩,栗永泽,方宝富.基于局部特征聚类损失和多类特征融合的面部表情识别.模式识别与人工智能, 2020, 33(3): 268-276.
(WANG H, LI Y Z, FANG B F.Locality Feature Aggregation Loss and Multi-feature Fusion for Facial Expression Recognition. Pattern Recognition and Artificial Intelligence, 2020, 33(3): 268-276.)
[7] LANITIS A, TAYLOR C J, COOTES T F.Automatic Interpretation and Coding of Face Images Using Flexible Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 743-756.
[8] TIAN Y I, KANADE T, COHN J F.Recognizing Action Units for Facial Expression Analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 97-115.
[9] ZHANG L G, TJONDRONEGORO D, CHANDRAN V.Random Gabor Based Templates for Facial Expression Recognition in Images with Facial Occlusion. Neurocomputing, 2014, 145: 451-464.
[10] SHAN C F, GONG S G, MCOWAN P W.Facial Expression Re-cognition Based on Local Binary Patterns: A Comprehensive Study. Image and Vision Computing, 2009, 27(6): 803-816.
[11] ISLAM M S, AUWATANAMONGKOL S.Facial Expression Re-cognition Using Local Arc Pattern. Trends in Applied Sciences Research, 2014, 9(2): 113-120.
[12] 徐振国,张冠文,孟祥增,等.基于深度学习的学习者情感识别与应用.电化教育研究, 2019, 40(2): 87-94.
(XU Z G, ZHANG G W, MENG X Z, et al. Learners' Emotion Recognition and Its Application Based on Deep Learning. E-Education Research, 2019, 40(2): 87-94.)
[13] LECUN Y, BENGIO Y, HINTON G. Deep Learning.Nature, 2015, 521: 436-444.
[14] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-09-25]. https://arxiv.org/pdf/1409.1556.pdf.
[15] ZHANG C S, WANG P Y, CHEN K, et al. Identity-Aware Convolutional Neural Network for Facial Expression Recognition. Journal of Systems Engineering and Electronics, 2017, 28(4): 784-792.
[16] LI Y, ZENG J B, SHAN S G, et al. Occlusion Aware Facial Expression Recognition Using CNN with Attention Mechanism. IEEE Transactions on Image Processing, 2019, 28(5): 2439-2450.
[17] ZHAO Z Q, LIU Q S, WANG S M.Learning Deep Global Multi-scale and Local Attention Features for Facial Expression Recognition in the Wild. IEEE Transactions on Image Processing, 2021, 30: 6544-6556.
[18] XIE S Y, HU H F, CHEN Y Z.Facial Expression Recognition with Two-Branch Disentangled Generative Adversarial Network. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(6): 2359-2371.
[19] SEO M, LEE J, PARK J, et al. Sequential Feature Filtering Cla-ssifier. IEEE Access, 2021, 9: 97068-97078.
[20] 王莹莹. 基于卷积神经网络的人脸表情识别问题的研究.博士学位论文.济南:山东大学, 2020.
(WANG Y Y.Research on Facial Expression Recognition Based on Convolutional Neutral Network. Ph.D. Dissertation. Jinan,China: Shandong University, 2020.)
[21] VASWANI A, SHAZEER N, PARMAR N, et al.Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2017: 6000-6010.
[22] MOLLAHOSSEINI A, CHAN D, MAHOOR M H.Going Deeper in Facial Expression Recognition Using Deep Neural Networks // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington,USA: IEEE, 2016. DOI: 10.1109/WACV.2016.7477450.
[23] LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset(CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Wa-shington,USA: IEEE, 2010: 94-101.
[24] LI S, DENG W H.Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition. IEEE Transactions on Image Processing, 2019, 28(1): 356-370.
[25] ERDEM C E, TURAN C, AYDIN Z.BAUM-2: A Multilingual Audio-Visual Affective Face Database. Multimedia Tools and Applications, 2015, 74: 7429-7459.
[26] YI D, LEI Z, LIAO S C, et al. Learning Face Representation from Scratch[C/OL]. [2021-09-25]. https://arxiv.org/pdf/1411.7923.pdf.
[27] ZHANG K P, ZHANG Z P, LI Z F, et al. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503.
[28] MOLLAHOSSEINI A, HASANI B, MAHOOR M H.AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31.
[29] HAPPY S L, ROUTRAY A.Robust Facial Expression Classification Using Shape and Appearance Features // Proc of the 8th International Conference on Advances in Pattern Recognition. Washington,USA: IEEE, 2015. DOI: 10.1109/ICAPR.2015.7050661.
[30] LIU Z W, LI S, DENG W H.Boosting-POOF: Boosting Part Based One vs One Feature for Facial Expression Recognition in the Wild // Proc of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Washington,USA: IEEE, 2017: 967-972.
[31] YANG H Y, CIFTCI U, YIN L J.Facial Expression Recognition by De-expression Residue Learning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2018: 2168-2177.
[32] GU W F, VENKATESH Y V, XIANG C.A Novel Application of Self-Organizing Network for Facial Expression Recognition from Radial Encoded Contours. Soft Computing, 2010, 14(2): 113-122.
[33] GOYANI M, PATEL N.Recognition of Facial Expressions Using Local Mean Binary Pattern. Electronic Letters on Computer Vision and Image Analysis, 2017, 16(1): 54-67.
[34] FARAJZADEH N, PAN G, WU Z H.Facial Expression Recognition Based on Meta Probability Codes. Pattern Analysis and Applications, 2014, 17(4): 763-781.
[35] TURAN C, LAM K M.Histogram-Based Local Descriptors for Facial Expression Recognition(FER): A Comprehensive Study. Jour-nal of Visual Communication and Image Representation, 2018, 55: 331-341.
[36] LY S T, DO N T, LEE G S, et al. A 3D Face Modeling Approach for In-the-Wild Facial Expression Recognition on Image Datasets // Proc of the IEEE International Conference on Image Processing. Washington,USA: IEEE, 2019: 3492-3496.
[37] LI S T, GONG D Y, YUAN Y.Face Recognition Using Weber Local Descriptors. Neurocomputing, 2013, 122: 272-283.
[38] MOHAMMAD T, ALI M L.Robust Facial Expression Recognition Based on Local Monotonic Pattern(LMP) // Proc of the 14th International Conference on Computer and Information Technology. Washington,USA: IEEE, 2011: 572-576.
[39] TURAN C, LAM K M, HE X J.Soft Locality Preserving Map (SLPM) for Facial Expression Recognition[C/OL]. [2021-09-25].https://arxiv.org/ftp/arxiv/papers/1801/1801.03754.pdf.