1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601; 2. School of Software, Hefei University of Technology, Hefei 230601; 3. School of Computer Science and Technology, Anhui University, Hefei 230601
Abstract Lipreading is a technology that translates the silent video of a single speaker's lip motion into text. Due to the small amplitude of lip movements, the feature differentiation ability and the generalization ability of the model are both weak. To address this issue, the purification of lipreading visual features is studied from three dimensions including time, space and channel. A method for lipreading based on multiple visual attention network(LipMVA) is proposed. Firstly, channel-level features are calibrated adaptively by channel attention to mitigate the interference from meaningless channels. Then, two spatio-temporal attention modules with different granularities are employed to suppress the effect of unimportant pixels or frames. Finally, experiments on CMLR and GRID datasets demonstrate LipMVA can reduce the error rate and therefore its effectiveness is verified.
Fund:National Natural Science Foundation of China(No.62272143), Anhui Provincial Major Science and Technology Project(No.202203a05020025), University Synergy Innovation Program of Anhui Province(No.GXXT-2022-054), The Se-venth Special Support Plan for Innovation and Entrepreneurship in Anhui Province
Corresponding Authors:
XUE Feng, Ph.D., professor. His research interests include artificial intelligence, multimedia analysis and recommendation system.
About author:: XIE Yincen, Master student. His research interests include computer vision.CAO Mingwei, Ph.D., associate profe-ssor. His research interests include 3D reconstruction and virtual reality.
[1] CHUNG J S, SENIOR A, VINYALS O, et al. Lip Reading Sentences in the Wild // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2017: 6447-6456. [2] ZHAO Y, XU R, SONG M L.A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading // Proc of the 1st ACM Multimedia in Asia. New York, USA: ACM, 2019. DOI: 10.1145/3338533.3366579. [3] ZHANG X B, GONG H G, DAI X L, et al. Understanding Pictograph with Facial Features: End-to-End Sentence-Level Lip Reading of Chinese // Proc of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI, 2019: 9211-9218. [4] ZHAO Y, XU R, WANG X C, et al. Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers // Proc of the 34th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI, 2020: 6917-6924. [5] SIMONYAN K, ZISSERMAN A.Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2023-08-22].https://arxiv.org/abs/1409.1556. [6] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [7] ASSAEL Y M, SHILLINGFORD B, WHITESON S, et al. LipNet: End-to-End Sentence-Level Lipreading[C/OL].[2023-08-22]. https://arxiv.org/pdf/1611.01599.pdf. [8] XU K, LI D W, CASSIMATIS N, et al. LCANet: End-to-End Lipreading with Cascaded Attention-CTC // Proc of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Washington, USA: IEEE, 2018: 548-555. [9] XUE F, YANG T, LIU K, et al. LCSNet: End-to-End Lipreading with Channel-Aware Feature Selection. ACM Transactions on Multimedia Computing, Communications, and Applications, 2023. DOI: 10.1145/3524620. [10] TRAN D, BOURDEV L, FERGUS R, et al. Learning Spatiotemporal Features with 3D Convolutional Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 4489-4497. [11] AFOURAS T, CHUNG J S, SENIOR A, et al. Deep Audio-Visual Speech Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 44(12): 8717-8727. [12] PETRIDIS S, STAFYLAKIS T, MA P C, et al. Audio-Visual Spee-ch Recognition with a Hybrid CTC/Attention Architecture // Proc of the IEEE Spoken Language Technology Workshop. Washington, USA: IEEE, 2018: 513-520. [13] SUTSKEVER I, VINYALS O, LE Q V.Sequence to Sequence Learning with Neural Networks // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2014, II: 3104-3112. [14] MA P C, PETRIDIS S, PANTIC M.Visual Speech Recognition for Multiple Languages in the Wild. Nature Machine Intelligence, 2022, 4: 930-939. [15] VASWANI A, SHAZEER N, PARMAR N, et al.Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010. [16] GRAVES A, FERNáNDEZ S, GOMEZ F J, et al. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks // Proc of the 23rd International Conference on Machine learning. New York, USA: ACM, 2006: 369-376. [17] COOKE M, BARKER J, CUNNINGHAM S P, et al. An Audio-Visual Corpus for Speech Perception. An Audio-Visual Corpus for Speech Perception and Automatic. The Journal of the Acoustical Society of America, 2006, 120(5): 2421-2424. [18] GRAVES A.Sequence Transduction with Recurrent Neural Networks[C/OL]. [2023-08-22]. https://arxiv.org/pdf/1211.3711.pdf. [19] SRIVASTAVA R K, GREFF K, SCHMIDHUBER J.Highway Net-works[C/OL]. [2023-08-22].https://arxiv.org/pdf/1505.00387.pdf. [20] LI X, WANG W H, HU X L, et al. Selective Kernel Networks[C/OL].[2023-08-22]. https://arxiv.org/pdf/1903.06586.pdf. [21] CHEN W C, TAN X, XIA Y C, et al. DualLip: A System for Joint Lip Reading and Generation // Proc of the 28th ACM International Conference on Multimedia. New York, USA: ACM, 2020: 1985-1993. [22] HUANG Y Y, LIANG X F, FANG C W.CALLip: Lipreading Using Contrastive and Attribute Learning // Proc of the 29th ACM International Conference on Multimedia. New York, USA: ACM, 2021: 2492-2500. [23] CHEN T, KORNBLITH S, NOROUZI M, et al. A Simple Framework for Contrastive Learning of Visual Representations // Proc of the 37th International Conference on Machine Learning. New York, USA: ACM, 2020: 1597-1607. [24] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-Excitation Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 7132-7141. [25] WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional Block Attention Module // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer,2018: 3-19. [26] RAO S, RAHMAN T, ROCHAN M, et al. Video-Based Person Re-identification Using Spatial-Temporal Attention Networks[C/OL].[2023-08-22]. https://arxiv.org/abs/1810.11261. [27] WANG Z W, SHE Q, SMOLIC A.ACTION-Net: Multipath Excitation for Action Recognition // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 13209-13218. [28] BAHDANAU D, CHO K, BENGIO Y.Neural Machine Translation by Jointly Learning to Align and Translate[C/OL]. [2023-08-22].https://arxiv.org/abs/1409.0473. [29] KINGMA D P, BA J L.Adam: A Method for Stochastic Optimization[C/OL]. [2023-08-22].https://arxiv.org/abs/1412.6980. [30] BENGIO S, VINYALS O, JAITLY N, et al.Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks // Proc of the 28th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2015: 1171-1179. [31] XUE F, LI Y, LIU D Y, et al. LipFormer: Learning to Lipread Unseen Speakers Based on Visual-Landmark Transformers. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(9): 4507-4517. [32] ZHOU B L, KHOSLA A, LAPEDRIZA A, et al. Learning Deep Features for Discriminative Localization // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2921-2929.
2024, Vol. 37 
