特征增强与残差重塑的多重一致性约束半监督视频动作检测

doi:10.16451/j.cnki.issn1003-6059.202405002

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (1239 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要一致性正则化半监督视频动作检测方法对原始数据和增广数据进行特征表示时容易引起两类数据间判别域偏差,导致判别结果无法拟合.针对该问题,文中提出特征增强与残差重塑的多重一致性约束半监督视频动作检测方法.首先,将基础动作特征描述子在时空维进行连续性增强编码,获取视频动作理解中至关重要的上下文信息.然后,在通过残差特征重塑模块获得多尺度残差信息的同时进行特征重塑.为了降低不同数据间的判别偏差,分别从分类特征与动作定位特征角度对原始数据和增广数据施加多重一致性约束,实现模型对增广数据和原始数据判别结果和特征表示的匹配.最后,在JHMDB-21、UCF101-24数据集上的实验表明,文中方法能有效提高少样本标记条件下视频动作检测准确度,具有较强的竞争力.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	胡正平
	张琦明
	王雨露
	张和浩
	邸继锐

关键词 ：半监督学习, 视频动作检测, 特征增强, 多重一致性约束

Abstract：The feature representations of both original data and augmented data in the consistency regularized semi-supervised video action detection method tend to induce discriminative domain bias between two types of data, thereby resulting in inadequate fitting of the discriminative results. To address this issue, a multi-consistency constrained semi-supervised video action detection method based on feature enhancement and residual reshaping is proposed in this paper. Firstly, the basic action feature descriptors are continuously enhanced and encoded in the spatiotemporal dimension to obtain crucial contextual information for video action understanding. Subsequently, a residual feature reshaping module is employed to obtain multi-scale residual information while reshaping the features. To reduce the discriminative bias between different types of data, multiple consistency constraints are applied to the original data and the augmented data from the perspectives of classification features and action localization features, achieving a match between discriminative results and feature representation of the augmented data and the original data. Experimental results on JHMDB-21 and UCF101-24 datasets demonstrate the effectiveness of the proposed method in improving video action detection accuracy under the condition of limited labeled samples and strong competitiveness.

Key words： Semi-supervised Learning Video Action Detection Feature Enhancement Multiple Consistency Constraints

收稿日期: 2024-04-03

ZTFLH:

TP391.41

基金资助:国家自然科学基金项目(No.61771420)、国家自然科学基金青年科学基金项目(No.62001413)资助

通讯作者: 胡正平,博士,教授,主要研究方向为模式识别、视频处理.E-mail:hzp@ysu.edu.cn.

作者简介: 张琦明,硕士研究生,主要研究方向为半监督视频动作检测.E-mail:zhangqiming@stumail.ysu.edu.cn.王雨露,硕士研究生,主要研究方向为基于骨骼的人体动作识别.E-mail:hiwangyulu@163.com.张和浩,博士研究生,主要研究方向为3D人体姿态估计.E-mail:zhanghh@stumail.ysu.edu.cn.邸继锐,博士研究生,主要研究方向为细粒度动作识别.E-mail:dijirui123@163.com.

引用本文:

胡正平, 张琦明, 王雨露, 张和浩, 邸继锐. 特征增强与残差重塑的多重一致性约束半监督视频动作检测[J]. 模式识别与人工智能, 2024, 37(5): 398-409. HU Zhengping, ZHANG Qiming, WANG Yulu, ZHANG Hehao, DI Jirui. Multi-consistency Constrained Semi-supervised Video Action Detection Based on Feature Enhancement and Residual Reshaping. Pattern Recognition and Artificial Intelligence, 2024, 37(5): 398-409.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202405002 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2024/V37/I5/398

[1] GU J X, WANG Z H, KUEN J, et al. Recent Advances in Convo-lutional Neural Networks. Pattern Recognition, 2018, 77(1): 354-377.
[2] LECUN Y, BENGIO Y, HINTON G. Deep Learning. Nature, 2015, 521(7553): 436-444.
[3] DEMIR U, RAWAT Y S, SHAH M. TinyVIRAT: Low-Resolution Video Action Recognition // Proc of the 25th International Confe-rence on Pattern Recognition. Washington, USA: IEEE, 2021: 7387-7394.
[4] KIZILTEPE R S, GAN J Q, ESCOBAR J J. Integration of Feature and Decision Fusion with Deep Learning Architectures for Video Classification. IEEE Access, 2024, 12: 19432-19446.
[5] YOSHIKAWA Y, SHIGETO Y, SHIMBO M, et al. Action Class Relation Detection and Classification Across Multiple Video Datasets. Pattern Recognition Letters, 2023, 173: 93-100.
[6] TIRUPATTUR P, DUARTE K, RAWAT Y S, et al. Modeling Multi-label Action Dependencies for Temporal Action Localization // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2021: 1460-1470.
[7] MU R H, MARCOLINO L, NI Q, et al. Enhancing Robustness in Video Recognition Models: Sparse Adversarial Attacks and Beyond. Neural Networks, 2024, 171: 127-143.
[8] VYAS S, RAWAT Y S, SHAH M. Multi-view Action Recognition Using Cross-View Video Prediction // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 427-444.
[9] JHUANG H, GALL J, ZUFFI S, et al. Towards Understanding Action Recognition // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2013: 3192-3199.
[10] KAY W, CARREIRA J, SIMONYAN K, et al. The Kinetics Human Action Video Dataset[C/OL].[2024-04-23]. https://arxiv.org/abs/1705.06950.
[11] SOOMRO K, ZAMIR A R, SHAH M. UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild[C/OL]. [2024-04-23].https://arxiv.org/abs/1212.0402.
[12] LEE D H. Pseudo-Label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks // Proc of the International Conference on Machine Learning Workshop. New York, USA: ACM, 2013: 896-902.
[13] HU S, LIU C H, DUTTA J, et al. PseudoProp: Robust Pseudo-Label Generation for Semi-supervised Object Detection in Autonomous Driving Systems // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2022: 4389-4397.
[14] ZHENG Z D, YANG Y. Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation. International Journal of Computer Vision, 2021, 129(4): 1106-1120.
[15] WANG Y C, WANG H C, SHEN Y J, et al. Semi-supervised Semantic Segmentation Using Unreliable Pseudo-Labels // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 4238-4247.
[16] CHANG H, XIE G C, YU J, et al. A Viable Framework for Semi-supervised Learning on Realistic Dataset. Machine Learning, 2023, 112(6): 1847-1869.
[17] LUO Y C, ZHU J, LI M X, et al. Smooth Neighbors on Teacher Graphs for Semi-supervised Learning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 8896-8905.
[18] BERTHELOT D, CARLINI N, GOODFEELOW I, et al. Mix-Ma-tch: A Holistic Approach to Semi-supervised Learning // Proc of the 33rd International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2019: 5050-5060.
[19] SAJJADI M, JAVANMARDI M, TASDIZEN T. Regularization with Stochastic Transformations and Perturbations for Deep Semi-supervised Learning // Proc of the 30th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2016: 1171-1179.
[20] GRANDVALET Y, BENGIO Y. Semi-supervised Learning by Entropy Minimization // Proc of the 17th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2004: 529-536.
[21] JEONG J, LEE S, KIM J, et al. Consistency-Based Semi-supervised Learning for Object Detection // Proc of the 33rd Internatio-nal Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2019: 10759-10768.
[22] FAN Y, KUKLEVA A, DAI D X, et al. Revisiting Consistency Regularization for Semi-supervised Learning. International Journal of Computer Vision, 2023, 131(3): 626-643.
[23] ZHONG X, YI A Y, LIU W X, et al. Background-Weakening Consistency Regularization for Semi-supervised Video Action Detection // Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington, USA: IEEE, 2023. DOI: 10.1109/ICASSP49357.2023.10095478.
[24] CARREIRA J, ZISSERMAN A. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 4724-4733.
[25] SIGURDSSON G A, VAROL G, WANG X L, et al. Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding // Proc of the 14th European Conference on Computer Vision. Berlin, Germany: Springer,2016: 510-526.
[26] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010.
[27] KUMAR A, RAWAT Y S. End-to-End Semi-supervised Learning for Video Action Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 14680-14690.
[28] ZHANG S W, SONG L, GAO C X, et al. GLNet: Global Local Network for Weakly Supervised Action Localization. IEEE Transactions on Multimedia, 2020, 22(10): 2610-2622.
[29] METTES P, SNOEK C G M, CHANG S F. Localizing Actions from Video Labels and Pseudo-annotations[C/OL].[2024-04-23]. https://arxiv.org/pdf/1707.09143v1.
[30] METTES P, SNOEK C G M. Pointly-Supervised Action Localization. International Journal of Computer Vision, 2019, 127(3): 263-281.
[31] CHÉRON G, ALAYRAC J B, LAPTEV I, et al. A Flexible Model for Training Action Localization with Varying Levels of Supervision // Proc of the 32nd International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2018: 950-961.
[32] ESCORCIA V, DAO C D, JAIN M, et al. Guess Where? Actor-Supervision for Spatiotemporal Action Localization. Computer Vision and Image Understanding, 2020, 192(1). DOI: 10.1016/j.cviu.2019.102886.
[33] ARNAB A, SUN C, NAGRANI A, et al. Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 751-768.
[34] SINGH G, SAHA S, SAPIENZA M, et al. Online Real-Time Multiple Spatiotemporal Action Localisation and Prediction // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2017: 3657-3666.
[35] HOU R, CHEN C, SHAH M. Tube Convolutional Neural Network(T-CNN) for Action Detection in Videos // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 5823-5832.
[36] KALOGEITON V, WEINZAEPFEL P, FERRARI V, et al. Action Tubelet Detector for Spatio-Temporal Action Localization // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4415-4423.
[37] DUARTE K, RAWAT Y S, SHAH M. VideoCapsuleNet: A Simplified Network for Action Detection // Proc of the 32nd International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2018: 7610-7619.
[38] SOGN L, ZHANG S W, YU G, et al. TACNet: Transition-Aware Context Network for Spatio-Temporal Action Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 11979-11987.
[39] ZHAO J J, SNOEK C G M. Dance with Flow: Two-in-One Stream Action Detection // Proc of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 9927-9936.
[40] LI Y X, WANG Z X, WANG L M, et al. Actions as Moving Points // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 68-84.
[41] MA X R, LUO Z G, ZHANG X, et al. Spatio-Temporal Action Detector with Self-Attention // Proc of the International Joint Conference on Neural Networks. Washington, USA: IEEE, 2021. DOI: 10.1109/IJCNN52387.2021.9533300.
[42] YANG J H, WANG K, LI R F, et al. Cascading Spatio-Temporal Attention Network for Real-Time Action Detection. Machine Vision and Applications, 2023, 34(6). DOI: 10.1007/s00138-023-01457-4.