Facial Expression Recognition Combining Self-Attention Feature Filtering Classifier and Two-Branch GAN
CHENG Yan1,2, CAI Zhuang1, WU Gang1, LUO Pin1, ZOU Haifeng1
1. School of Computer and Information Engineering, Jiangxi Normal University, Nanchang 330022; 2. Jiangxi Provincial Key Laboratory of Intelligent Education, Science and Technology Department of Jiangxi Province, Nan-chang 330022
Abstract The expression features extracted by the existing facial expression recognition methods are usually mixed with other facial attributes, which is not conducive to facial expression recognition. A facial expression recognition model combining self-attention feature filter classifier and two-branch generative adversarial network is proposed. Two-branch generative adversarial network is introduced to learn discriminative expression representation, and a self-attention feature filtering classifier is proposed as the expression classification module. The cascaded LayerNorm and ReLU are employed to zero the low activation unit and retain the high activation unit to generate multi-level features. The self-attention is utilized to fuse and output the prediction results of multi-level features, and consequently the influence of noise on the recognition results is eliminated to a certain extent. A sliding module based dual image consistency loss supervised model is proposed to learn discriminative expression representations. The reconstruction loss is calculated by a sliding window and more attention is paid to the details. Finally, experiments on CK+, RAF-DB, TFEID and BAUM-2i datasets show the proposed model achieves better recognition results.
Fund:National Natural Science Foundation of China(No.62167006,61967011), Key Project of National Social Science Fund of China(No.20AXW009), Natural Science Foundation of Jiangxi Province(No.20202BABL202033, 20212BAB202017), Key Project of Humanities and Social Sciences of Jiangxi Provincial Department of Education(No.JD19056), Jiangxi Province Science and Technology Innovation Base Plan-Jiangxi Province Key Laboratory Project (No.20212BCD42001), Jiangxi Province 03 Special and 5G Projects(No.20212ABC03A22), Jiangxi Province Major Disciplines Academic and Technical Leaders Training plan-Leading Talent Project(No.20213BCJL22047)
Corresponding Authors:CHENG Yan, Ph.D., professor. Her research interests include artificial intelligence, intelligent information processing and sentiment analysis.
About author:: CAI Zhuang, master student. His research interests include facial expression recognition and deep learning. WU Gang, master student. His research interests include deep learning and cognitive diagnosis. LUO Pin, master student. His research interests include deep learning and sentiment analysis. ZOU Haifeng, master student. His research interests include deep learning and sentiment analysis.
[1] MEHRABIAN A.Communication without Words // MORTENSEN C D, ed. Communication Theory. 2nd Edition. New York,USA: Routledge, 2008: 193-200. [2] LUCEY P, COHN J F, MATTHEWS L, et al. Automatically Detecting Pain in Video through Facial Action Units. IEEE Transactions on Systems, Man,and Cybernetics(Cybernetics), 2011, 41(3): 664-674. [3] 龚礼林,刘红霞,赵蔚,等. 情感导学系统(ATS)的关键技术及其导学模型研究——论智能导学系统走向情感导学系统之意蕴.远程教育杂志, 2019, 37(5): 45-55. (GONG L L, LIU H X, ZHAO W, et al. Research on Key Techniques of Affective Tutor System and Its Tutoring Model: The Implications from Intelligent Tutoring System to Affective Tutoring System. Journal of Distance Education, 2019, 37(5): 45-55.) [4] SCHRODER M, BEVACQUA E, COWIE R, et al. Building Auto-nomous Sensitive Artificial Listeners. IEEE Transactions on Affective Computing, 2012, 3(2): 165-183. [5] JEONG M, KO B C.Driver's Facial Expression Recognition in Real-Time for Safe Driving. Sensors, 2018, 18(12): 4270-4287. [6] 王浩,栗永泽,方宝富.基于局部特征聚类损失和多类特征融合的面部表情识别.模式识别与人工智能, 2020, 33(3): 268-276. (WANG H, LI Y Z, FANG B F.Locality Feature Aggregation Loss and Multi-feature Fusion for Facial Expression Recognition. Pattern Recognition and Artificial Intelligence, 2020, 33(3): 268-276.) [7] LANITIS A, TAYLOR C J, COOTES T F.Automatic Interpretation and Coding of Face Images Using Flexible Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 743-756. [8] TIAN Y I, KANADE T, COHN J F.Recognizing Action Units for Facial Expression Analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 97-115. [9] ZHANG L G, TJONDRONEGORO D, CHANDRAN V.Random Gabor Based Templates for Facial Expression Recognition in Images with Facial Occlusion. Neurocomputing, 2014, 145: 451-464. [10] SHAN C F, GONG S G, MCOWAN P W.Facial Expression Re-cognition Based on Local Binary Patterns: A Comprehensive Study. Image and Vision Computing, 2009, 27(6): 803-816. [11] ISLAM M S, AUWATANAMONGKOL S.Facial Expression Re-cognition Using Local Arc Pattern. Trends in Applied Sciences Research, 2014, 9(2): 113-120. [12] 徐振国,张冠文,孟祥增,等.基于深度学习的学习者情感识别与应用.电化教育研究, 2019, 40(2): 87-94. (XU Z G, ZHANG G W, MENG X Z, et al. Learners' Emotion Recognition and Its Application Based on Deep Learning. E-Education Research, 2019, 40(2): 87-94.) [13] LECUN Y, BENGIO Y, HINTON G. Deep Learning.Nature, 2015, 521: 436-444. [14] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-09-25]. https://arxiv.org/pdf/1409.1556.pdf. [15] ZHANG C S, WANG P Y, CHEN K, et al. Identity-Aware Convolutional Neural Network for Facial Expression Recognition. Journal of Systems Engineering and Electronics, 2017, 28(4): 784-792. [16] LI Y, ZENG J B, SHAN S G, et al. Occlusion Aware Facial Expression Recognition Using CNN with Attention Mechanism. IEEE Transactions on Image Processing, 2019, 28(5): 2439-2450. [17] ZHAO Z Q, LIU Q S, WANG S M.Learning Deep Global Multi-scale and Local Attention Features for Facial Expression Recognition in the Wild. IEEE Transactions on Image Processing, 2021, 30: 6544-6556. [18] XIE S Y, HU H F, CHEN Y Z.Facial Expression Recognition with Two-Branch Disentangled Generative Adversarial Network. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(6): 2359-2371. [19] SEO M, LEE J, PARK J, et al. Sequential Feature Filtering Cla-ssifier. IEEE Access, 2021, 9: 97068-97078. [20] 王莹莹. 基于卷积神经网络的人脸表情识别问题的研究.博士学位论文.济南:山东大学, 2020. (WANG Y Y.Research on Facial Expression Recognition Based on Convolutional Neutral Network. Ph.D. Dissertation. Jinan,China: Shandong University, 2020.) [21] VASWANI A, SHAZEER N, PARMAR N, et al.Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2017: 6000-6010. [22] MOLLAHOSSEINI A, CHAN D, MAHOOR M H.Going Deeper in Facial Expression Recognition Using Deep Neural Networks // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington,USA: IEEE, 2016. DOI: 10.1109/WACV.2016.7477450. [23] LUCEY P, COHN J F, KANADE T, et al. The Extended Cohn-Kanade Dataset(CK+): A Complete Dataset for Action Unit and Emotion-Specified Expression // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Wa-shington,USA: IEEE, 2010: 94-101. [24] LI S, DENG W H.Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition. IEEE Transactions on Image Processing, 2019, 28(1): 356-370. [25] ERDEM C E, TURAN C, AYDIN Z.BAUM-2: A Multilingual Audio-Visual Affective Face Database. Multimedia Tools and Applications, 2015, 74: 7429-7459. [26] YI D, LEI Z, LIAO S C, et al. Learning Face Representation from Scratch[C/OL]. [2021-09-25]. https://arxiv.org/pdf/1411.7923.pdf. [27] ZHANG K P, ZHANG Z P, LI Z F, et al. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503. [28] MOLLAHOSSEINI A, HASANI B, MAHOOR M H.AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31. [29] HAPPY S L, ROUTRAY A.Robust Facial Expression Classification Using Shape and Appearance Features // Proc of the 8th International Conference on Advances in Pattern Recognition. Washington,USA: IEEE, 2015. DOI: 10.1109/ICAPR.2015.7050661. [30] LIU Z W, LI S, DENG W H.Boosting-POOF: Boosting Part Based One vs One Feature for Facial Expression Recognition in the Wild // Proc of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Washington,USA: IEEE, 2017: 967-972. [31] YANG H Y, CIFTCI U, YIN L J.Facial Expression Recognition by De-expression Residue Learning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2018: 2168-2177. [32] GU W F, VENKATESH Y V, XIANG C.A Novel Application of Self-Organizing Network for Facial Expression Recognition from Radial Encoded Contours. Soft Computing, 2010, 14(2): 113-122. [33] GOYANI M, PATEL N.Recognition of Facial Expressions Using Local Mean Binary Pattern. Electronic Letters on Computer Vision and Image Analysis, 2017, 16(1): 54-67. [34] FARAJZADEH N, PAN G, WU Z H.Facial Expression Recognition Based on Meta Probability Codes. Pattern Analysis and Applications, 2014, 17(4): 763-781. [35] TURAN C, LAM K M.Histogram-Based Local Descriptors for Facial Expression Recognition(FER): A Comprehensive Study. Jour-nal of Visual Communication and Image Representation, 2018, 55: 331-341. [36] LY S T, DO N T, LEE G S, et al. A 3D Face Modeling Approach for In-the-Wild Facial Expression Recognition on Image Datasets // Proc of the IEEE International Conference on Image Processing. Washington,USA: IEEE, 2019: 3492-3496. [37] LI S T, GONG D Y, YUAN Y.Face Recognition Using Weber Local Descriptors. Neurocomputing, 2013, 122: 272-283. [38] MOHAMMAD T, ALI M L.Robust Facial Expression Recognition Based on Local Monotonic Pattern(LMP) // Proc of the 14th International Conference on Computer and Information Technology. Washington,USA: IEEE, 2011: 572-576. [39] TURAN C, LAM K M, HE X J.Soft Locality Preserving Map (SLPM) for Facial Expression Recognition[C/OL]. [2021-09-25].https://arxiv.org/ftp/arxiv/papers/1801/1801.03754.pdf.
2022, Vol. 35 
