Abstract Each individual makes facial expressions in a unique way. In this paper, a locality-feature-aggregation(LFA) loss function is proposed. Differences between images of the same class are reduced and those between images of different classes are expanded during the training of deep neural network. Thus, the influence of expression polymorphism on feature extraction by deep learning is weakened. The local areas with rich expressions can express facial expression features better. A deep learning network framework incorporating LFA loss function is proposed. Local features of facial images are extracted for facial expression recognition. Compared with other methods, the proposed method is more effective on real world RAF datasets and CK+ datasets under laboratory conditions.
Fund:Supported by National Natural Science Foundation of China(No.61872327), Fundamental Research Funds for Central Universities(No.ACAIM190102), Project of Innovation Team of Ministry of Education of China(No.IRT17R32), Natural Science Foundation of Anhui Province(No.1708085MF146), Project of Collaborative Innovation in Anhui Colleges and Universities(No.GXXT-2019-003)
Corresponding Authors:
FANG Baofu, Ph.D., professor. His research interests include machine learning, machine vision and multi-robot systems.
About author:: WANG Hao, Ph.D., professor. His research interests include intelligent computing theory and software,artificial intelligence, machine vision and data mining. LI Yongze, master student. His research interests include computer application techno-logy, facial expression recognition and machine learning.
WANG Hao,LI Yongze,FANG Baofu. Locality Feature Aggregation Loss and Multi-feature Fusion for Facial Expression Recognition[J]. , 2020, 33(3): 268-276.
[1] DARWIN C R, EKMAN P. The Expression of the Emotions in Man and Animals. Oxford, UK: Oxford University Press, 1998. [2] TIAN Y I, KANADE T, COHN J F. Recognizing Action Units for Facial Expression Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(2): 97-115. [3] MEHRABIAN A. Nonverbal Communication. New York, USA: Tran-saction Publishers, 1972. [4] LUCEY P, COHN J F, MATTHEWS I, et al. Automatically Detecting Pain in Video through Facial Action Units. IEEE Transactions on Systems, Man, and Cybernetics(Cybernetics), 2010, 41(3): 664-674. [5] MCDUFF D, EL KALIOUBY R, SENECHAL T, et al. Automatic Measurement of AD Preferences from Facial Responses Gathered over the Internet. Image and Vision Computing, 2014, 32(10): 630-640. [6] SCHRODER M, BEVACQUA E, COWIE R, et al. Building Autonomous Sensitive Artificial Listeners. IEEE Transactions on Affective Computing, 2012, 3(2): 165-183. [7] TSIAMYRTZIS P, DOWDALL J, SHASTRI D, et al. Imaging Facial Physiology for the Detection of Deceit. International Journal of Computer Vision, 2007, 71(2): 197-214. [8] BAO H, MA T. Feature Extraction and Facial Expression Recognition Based on Bezier Curve // Proc of the IEEE International Conference on Computer and Information Technology. Washington, USA: IEEE, 2014: 884-887. [9] ZHANG L G, TJONDRONEGORO D, CHANDRAN V. Random Gabor Based Templates for Facial Expression Recognition in Images with Facial Occlusion. Neurocomputing, 2014, 145: 451-464. [10] SHAN C F, GONG S G, MCOWAN P W. Facial Expression Re-cognition Based on Local Binary Patterns: A Comprehensive Study. Image and Vision Computing, 2009, 27(6): 803-816. [11] CHEN J H, TAKIGUCHI T, ARIKI Y. Rotation-Reversal Invariant HOG Cascade for Facial Expression Recognition. Signal, Image and Video Processing, 2017, 11(8): 1485-1492. [12] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Cla-ssification with Deep Convolutional Neural Networks // PEREIRA F, BURGES C J C, BOTTOU L, et al., eds. Advances in Neural Information Processing Systems 25. Cambridge, USA: The MIT Press, 2012: 1097-1105. [13] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2019-09-25]. https://128.84.21.199/pdf/1409.1556v2.pdf. [14] SZEGEDY C, LIU W, JIA Y Q, et al. Going Deeper with Convolutions // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015. DOI: 10.1109/CVPR.2015.7298594. [15] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [16] JUNG H, LEE S, YIM J, et al. Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 2983-2991. [17] ZHAO X Y, LIANG X D, LIU L Q, et al. Peak-Piloted Deep Network for Facial Expression Recognition // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 425-442. [18] YANG H Y, CIFTCI U, YIN L J. Facial Expression Recognition by De-expression Residue Learning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2168-2177. [19] HAPPY S L, ROUTRAY A. Automatic Facial Expression Reco-gnition Using Features of Salient Facial Patches. IEEE Transactions on Affective Computing, 2015, 6(1): 1-12. [20] MAJUMDER A, BEHERA L, SUBRAMANIAN V K. Emotion Recognition from Geometric Facial Features Using Self-organizing Map. Pattern Recognition, 2014, 47(3): 1282-1293. [21] ZHONG L, LIU Q S, YANG P, et al. Learning Multiscale Active Facial Patches for Expression Analysis. IEEE Transactions on Cybernetics, 2014, 45(8): 1499-1510. [22] WANG H, WEI S B, FANG B F. Facial Expression Recognition Using Iterative Fusion of MO-HOG and Deep Features. The Journal of Supercomputing, 2018. DOI: 10.1007/s11227-018-2554-8. [23] WEN Y D, ZHANG K P, LI Z F, et al. A Discriminative Feature Learning Approach for Deep Face Recognition // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 499-515. [24] LI S, DENG W H. Reliable Crowd Sourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recogni-tion. IEEE Transactions on Image Processing, 2019, 28(1): 356-370. [25] FAN Y R, LAM J C K, LI V O K. Multi-region Ensemble Convolutional Neural Network for Facial Expression Recognition // Proc of the International Conference on Artificial Neural Networks. Berlin, Germany: Springer, 2018: 84-94. [26] ZENG J B, SHAN S G, CHEN X L. Facial Expression Recognition with Inconsistently Annotated Datasets // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 227-243. [27] ACHARYA D, HUANG Z W, PAUDEL D P, et al. Covariance Pooling for Facial Expression Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 480-487. [28] LIU P, HAN S Z, MENG Z B, et al. Facial Expression Recognition via a Boosted Deep Belief Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2014: 1805-1812. [29] LIU M Y, LI S X, SHAN S G, et al. Au-inspired Deep Networks for Facial Expression Feature Learning. Neurocomputing, 2015, 159: 126-136. [30] KAZEMI V, SULLIVAN J. One Millisecond Face Alignment with an Ensemble of Regression Trees // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2014: 1867-1874. [31] MOLLAHOSSEINI A, CHAN D, MAHOOR M H. Going Deeper in Facial Expression Recognition Using Deep Neural Networks // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2016. DOI: 10.1109/WACV.2016.7477450. [32] LEE S H, BADDAR W J, RO Y M. Collaborative Expression Re-presentation Using Peak Expression and Intra Class Variation Face Images for Practical Subject-Independent Emotion Recognition in Videos. Pattern Recognition, 2016, 54: 52-67. [33] LOPES A T, DE AGUIAR E, DE SOUZA A F, et al. Facial Expression Recognition with Convolutional Neural Networks: Coping with Few Data and the Training Sample Order. Pattern Recognition, 2017, 61: 610-628.
2020, Vol. 33 
