Abstract The existing micro-expression recognition technologies cannot make full use of the spatiotemporal features near the apex frame. Aiming at this problem, a micro-expression recognition algorithm based on 3D convolutional neural network and optical flow fields from the neighboring frames of the apex frame is proposed. Firstly, the optical flow fields between the adjacent frames before and after the apex frame are extracted. The important spatiotemporal information of micro-expressions are retained while the redundant information is removed and the computation load is reduced. Then, a 3D convolutional neural network is employed to extract the enhanced spatiotemporal features from the optical flow fields and thus the classification is completed. Finally, experiments on three spontaneous micro-expression databases show the proposed algorithm produces a better accuracy.
ZHANG Xuesen,JIA Jingping. Micro-Expression Recognition Algorithm Based on 3D Convolutional Neural Network and Optical Flow Fields from Neighboring Frames of Apex Frame[J]. , 2021, 34(5): 423-433.
[1] YAN W J,WU Q,LIANG J,et al. How Fast Are the Leaked Facial Expressions:The Duration of Micro-Expressions.Journal of Nonverbal Behavior,2013,37(4):217-230. [2] MATSUMOTO D,HWANG H S.Evidence for Training the Ability to Read Microexpressions of Emotion.Motivation and Emotion,2011,35(2):181-191. [3] TURNER J H.The Evolution of Emotions//JAN STETS E,TU-RNER J H,eds.Handbook of the Sociology of Emotions:Volume II.Berlin,Germany:Springer,2014:11-13. [4] GAN Y S,LIONG S T.Bi-directional Vectors from Apex in CNN for Micro-Expression Recognition//Proc of the 3rd IEEE International Conference on Image,Vision and Computing.Washington,USA:IEEE,2018:168-172. [5] PENG M,WANG C Y,CHEN T,et al. Dual Temporal Scale Con-volutional Neural Network for Micro-Expression Recognition.Frontiers in Psychology,2017,8.DOI:10.3389/fpsyg.2017.01745.eCollection 2017. [6] PENG M,WU Z,ZHANG Z H,et al. From Macro to Micro Expression Recognition:Deep Learning on Small Datasets Using Transfer Learning//Proc of the 13th IEEE International Conference on Automatic Face and Gesture Recognition.Washington,USA:IEEE,2018:657-661. [7] PFISTER T,LI X B,ZHAO G Y,et al. Recognising Spontaneous Facial Micro-Expressions//Proc of the International Conference on Computer Vision.Washington,USA:IEEE,2011:1449-1456. [8] WANG Y D,SEE J,PHAN R C W,et al. LBP with Six Intersection Points:Reducing Redundant Information in LBP-Top for Micro-expression Recognition//Proc of the Asian Conference on Computer Vision.Berlin,Germany:Springer,2014:525-537. [9] LI Y T,HUANG X H,ZHAO G Y.Can Micro-Expression Be Re-cognized Based on Single Apex Frame?//Proc of the 25th IEEE International Conference on Image Processing.Washington,USA:IEEE,2018:3094-3098. [10] GAN Y S,LIONG S T,YAU W C,et al.OFF-ApexNet on Micro-Expression Recognition System.Signal Processing:Image Communication,2019,74:129-139. [11] POLIKOVSKY S,KAMEDA Y,OHTA Y.Facial Micro-Expre-ssions Recognition Using High Speed Camera and 3D-Gradient Descriptor//Proc of the 3rd International Conference on Imaging for Crime Detection and Prevention.New York,USA:ACM,2009.DOI:10.1049/ic.2009.0244. [12] QU F B,WANG S J,YAN W J,et al.CAS(ME)2:A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition.IEEE Transactions on Affective Computing,2017,9(4):424-436. [13] WANG Y D,SEE J,PHAN R C W,et al.Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expre-ssion Recognition.PloS One,2015,10(5):e0124674. [14] ESMAEILI V,SHAHDI S O.Automatic Micro-Expression Apex Spotting Using Cubic-LBP.Multimedia Tools and Applications,2020,79:20221-20239. [15] LI X B,HONG X P,MOILANEN A,et al.Towards Reading Hi-dden Emotions:A Comparative Study of Spontaneous Micro-Expression Spotting and Recognition Methods.IEEE Transactions on Affective Computing,2018,9(4):563-577. [16] ZHANG P,BEN X Y,YAN R,et al.Micro-Expression Recognition System.Optik,2016,127(3):1395-1400. [17] LIN C H,LONG F,HUANG J M,et al.Micro-Expression Recognition Based on Spatiotemporal Gabor Filters//Proc of the 8th International Conference on Information Science and Technology.Washington,USA:IEEE,2018:487-491. [18] LIU Y J,ZHANG J K,YAN W J,et al. A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition.IEEE Transactions on Affective Computing,2016,7(4):299-310. [19] XU F,ZHANG J P,WANG J Z.Microexpression Identification and Categorization Using a Facial Dynamics Map.IEEE Transactions on Affective Computing,2017,8(2):254-267. [20] MANOHAR V,SHREVE M,GOLDGOF D,et al. Finite Element Modeling of Facial Deformation in Videos for Computing Strain Pa-ttern//Proc of the 19th International Conference on Pattern Re-cognition.Washington,USA:IEEE,2008.DOI:10.1109/ICPR.2008.4761314. [21] SHREVE M,GODAVARTHY S,MANOHAR V,et al. Towards Macro-and Micro-Expression Spotting in Video Using Strain Pa-tterns//Proc of the Workshop on Applications of Computer Vision.Washington,USA:IEEE,2009.DOI:10.1109/WACV.2009.5403044. [22] LIONG S T,SEE J,WONG K S,et al.Less Is More:Micro-Expression Recognition from Video Using Apex Frame.Signal Processing:Image Communication,2018,62:82-92. [23] XIA Z Q,HONG X P,GAO X Y,et al. Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-Expre-ssions.IEEE Transactions on Multimedia,2020,22(3):626-640. [24] PATEL D,HONG X P,ZHAO G Y.Selective Deep Features for Micro-Expression Recognition//Proc of the 23rd International Conference on Pattern Recognition.Washington,USA:IEEE,2016:2258-2263. [25] KHOR H Q,SEE J,PHAN R C W,et al.Enriched Long-Term Recurrent Convolutional Network for Facial Micro-Expression Re-cognition//Proc of the 13th IEEE International Conference on Automatic Face and Gesture Recognition.Washington,USA:IEEE,2018:667-674. [26] REDDY S P T,TEJA K S,DUBEY S R,et al. Spontaneous Facial Micro-Expression Recognition Using 3D Spatiotemporal Convolutional Neural Networks//Proc of the International Joint Confe-rence on Neural Networks.Washington,USA:IEEE,2019.DOI:10.1109/IJCNN.2019.8852419. [27] LI J,WANG Y D,SEE J,et al.Micro-Expression Recognition Based on 3D Flow Convolutional Neural Network.Pattern Analysis and Applications,2019,22(4):1331-1339. [28] XIA Z Q,ZHANG W H,TAN F,et al.An Accurate Eye Localization Approach for Smart Embedded System//Proc of the 6th International Conference on Image Processing Theory,Tools and Applications.Washington,USA:IEEE,2016.DOI:10.1109/IPTA.2016.7821006. [29] XIA Z Q,FENG X Y,PENG J Y,et al.Spontaneous Micro-Expression Spotting via Geometric Deformation Modeling.Computer Vision and Image Understanding,2016,147:87-94. [30] FARNEBÄCK G.Two-Frame Motion Estimation Based on Polynomial Expansion//Proc of the Scandinavian Conference on Image Analysis.Berlin,Germany:Springer,2003:363-370. [31] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet Cla-ssification with Deep Convolutional Neural Networks//Proc of the 25th International Conference on Neural Information Processing Systems.Cambridge,USA:The MIT Press,2012,I:1097-1105. [32] BOUREAU Y L,PONCE J,LECUN Y.A Theoretical Analysis of Feature Pooling in Visual Recognition//Proc of the 27th International Conference on Machine Learning.New York,USA:ACM,2010:111-118. [33] HUANG X H,WANG S J,LIU X,et al.Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition.IEEE Transactions on Affective Computing,2019,10(1):32-47. [34] YAN W J,LI X B,WANG S J,et al. CASME II:An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation.PloS One,2014,9(1):e86041. [35] LI X B,PFISTER T,HUANG X H,et al.A Spontaneous Micro-Expression Database:Inducement,Collection and Baseline//Proc of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition.Washington,USA:IEEE,2013.DOI:10.1109/FG.2013.6553717. [36] DAVISON A K,LANSLEY C,COSTEN N,et al.SAMM:A Spontaneous Micro-Facial Movement Dataset.IEEE Transactions on Affective Computing,2018,9(1):116-129. [37] SEE J,YAP M H,LI J T,et al.MEGC 2019-the Second Facial Micro-Expressions Grand Challenge//Proc of the 14th IEEE International Conference on Automatic Face and Gesture Recognition.Washington,USA:IEEE,2019.DOI:10.1109/FG.2019.8756611. [38] LIONG S T,WONG K S.Micro-Expression Recognition Using Apex Frame with Phase Information//Proc of the Asia-Pacific Signal and Information Processing Association Annual Summit and Confe-rence.Washington,USA:IEEE,2017:534-537. [39] LIONG S T,SEE J,WONG K S,et al.Automatic Micro-Expre-ssion Recognition from Long Video Using a Single Spotted Apex//Proc of the Asian Conference on Computer Vision.Berlin,Germany:Springer,2016:345-360. [40] WANG C Y,PENG M,BI T,et al.Micro-attention for Micro-Expression Recognition.Neurocomputing,2020,410:354-362. [41] ayush2610:Microfacial-exp[C/OL].[2020-10-05].https://github.com/ayush2610/Microfacial-exp. [42] VAN QUANG N,CHUN J,TOKUYAMA T.CapsuleNet for Micro-Expression Recognition//Proc of the 14th IEEE International Conference on Automatic Face and Gesture Recognition.Washington,USA; IEEE,2019.DOI:10.1109/FG.2019.8756544.
2021, Vol. 34 
