Review of Deep Learning-Based Video Anomaly Detection
JI Genlin1, QI Xiaosha2, WANG Jiaqi2,3
1. School of Computer and Electronic Information/School of Arti-ficial Intelligence, Nanjing Normal University, Nanjing 210023; 2. School of Mathematical Sciences, Nanjing Normal University, Nanjing 210023; 3. School of Foreign Languages and Cultures, Nanjing Normal University, Nanjing 210023
Abstract The study of video anomaly detection involves the methods such as probabilistic statistics, machine learning and deep learning. The purpose of this paper is to synthesize the research results of the author's group and other advanced researches with a focus on deep learning-based video anomaly detection methods, comprehensively discussing the background, challenges and solutions in this field. Most relevant papers in the field are synthesized and systematically analyzed to provide the scholars with a fundamental understanding of the current research progress. The deep learning-based video anomaly detection methods are classified and analyzed. The network model selection for different methods is summarized. The commonly used datasets and performance evaluation indexes are introduced in detail. The advantages of various methods are highlighted by the performance comparison, and the future research directions and application scenarios in the field of video anomaly detection are deeply explored and forecasted.
Fund:National Natural Science Foundation of China(No.41971343)
Corresponding Authors:
JI Genlin, Ph.D., professor. His research interests include big data analysis and mining technology.
About author:: QI Xiaosha, Ph.D. candidate. Her research interests include big data analysis and video anomaly detection. WANG Jiaqi, Ph.D. candidate. Her research interests include big data analysis and video anomaly detection.
JI Genlin,QI Xiaosha,WANG Jiaqi. Review of Deep Learning-Based Video Anomaly Detection[J]. Pattern Recognition and Artificial Intelligence, 2024, 37(2): 128-143.
[1] LIU W J, CAO J X, ZHU Y L, et al. Real-Time Anomaly Detection on Surveillance Video with Two-Stream Spatio-Temporal Generative Model. Multimedia Systems, 2023, 29(1): 59-71. [2] ABERKANE S, ELARBI-BOUDIHIR M.Deep Reinforcement Lear-ning-Based Anomaly Detection for Video Surveillance. Informatica, 2022, 46: 291-298. [3] 胡正平,赵梦瑶,辛丙一.结合全局与局部视频表示的视频异常检测算法.模式识别与人工智能, 2020, 33(2): 133-140. (HU Z P, ZHAO M Y, XIN B Y.Video Anomaly Detection Algorithm Combining Global and Local Video Representation. Pattern Recognition and Artificial Intelligence, 2020, 33(2): 133-140.) [4] KIRAN B R, THOMAS D M, PARAKKAL R.An Overview of Deep Learning Based Methods for Unsupervised and Semi-supervised Anomaly Detection in Videos. Journal of Imaging, 2018, 4(2). DOI: 10.3390/jimaging4020036. [5] 胡正平,张乐,李淑芳,等.视频监控系统异常目标检测与定位综述.燕山大学学报, 2019, 43(1): 1-12. (HU Z P, ZHANG L, LI S F, et al. Review of Abnormal Behavior Detection and Location for Intelligent Video Surveillance Systems. Journal of Yanshan University, 2019, 43(1): 1-12.) [6] ZAHEER M Z, LEE J H, LEE S I, et al. A Brief Survey on Contemporary Methods for Anomaly Detection in Videos//Proc of the International Conference on Information and Communication Technology Convergence. Washington, USA: IEEE, 2019: 472-473. [7] MABROUK A B, ZAGROUBA E.Abnormal Behavior Recognition for Intelligent Video Surveillance Systems: A Review. Expert Systems with Applications, 2018, 91(C): 480-491. [8] DHIMAN C, VISHWAKARMA D K.A Review of State-of-the-Art Techniques for Abnormal Human Activity Recognition. Engineering Applications of Artificial Intelligence, 2019, 77: 21-45. [9] AFIQ A A, ZAKARIYA M A, SAAD M N, et al. A Review on Classifying Abnormal Behavior in Crowd Scene. Journal of Visual Communication and Image Representation, 2019, 58: 285-303. [10] 王志国,章毓晋.监控视频异常检测:综述.清华大学学报(自然科学版), 2020, 60(6): 518-529. (WANG Z G, ZHANG Y J.Anomaly Detection in Surveillance Videos: A Survey. Journal of Tsinghua University (Science and Technology), 2020, 60(6): 518-529.) [11] 王思齐,胡婧韬,余广,等.智能视频异常事件检测方法综述.计算机工程与科学, 2020, 42(8): 1393-1405. (WANG S Q, HU J T, YU G, et al. A Survey of Video Abnormal Event Detection. Computer Engineering and Science, 2020, 42(8): 1393-1405.) [12] 吉根林,许振,李欣璐,等.监控视频中异常事件检测技术研究进展.南京航空航天大学学报, 2020, 52(5): 685-694. (JI G L, XU Z, LI X L, et al. Progress on Abnormal Event Detection Technology in Video Surveillance. Journal of Nanjing University of Aeronautics and Astronautics, 2020, 52(5): 685-694.) [13] 杨帆,肖斌,於志文.监控视频的异常检测与建模综述.计算机研究与发展, 2021, 58(12): 2708-2723. (YANG F, XIAO B, YU Z W.Anomaly Detection and Modeling of Surveillance Video. Journal of Computer Research and Development, 2021, 58(12): 2708-2723.) [14] REZAEE K, REZAKHANI S M, KHOSRAVI R M, et al. A Survey on Deep Learning-Based Real-Time Crowd Anomaly Detection for Secure Distributed Video Surveillance. Personal and Ubiquitous Computing, 2021. DOI: 10.1007/s00779-021-01586-5. [15] 徐涛,田崇阳,刘才华.基于深度学习的人群异常行为检测综述.计算机科学, 2021, 48(9): 125-134. (XU T, TIAN C Y, LIU C H.Deep Learning for Abnormal Crowd Behavior Detection: A Review. Computer Science, 2021, 48(9): 125-134.) [16] MU H Y, SUN R Z, YUAN G, et al. Abnormal Human Behavior Detection in Videos: A Review. Information Technology and Control, 2021, 50(3): 522-545. [17] 何平,李刚,李慧斌.基于深度学习的视频异常检测方法综述.计算机工程与科学, 2022, 44(9): 1620-1629. (HE P, LI G, LI H B.A Survey on Deep Learning Based Video Anomaly Detection. Computer Engineering and Science, 2022, 44(9): 1620-1629.) [18] PATRIKAR D R, PARATE M R.Anomaly Detection Using Edge Computing in Video Surveillance System: Review. International Journal of Multimedia Information Retrieval, 2022, 11(2): 85-110. [19] RAMACHANDRA B, JONES M J, VATSAVAI R R.A Survey of Single-Scene Video Anomaly Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2293-2312. [20] JEBUR S A, HUSSEIN K A, HOOMOD H K, et al. Review on Deep Learning Approaches for Anomaly Event Detection in Video Surveillance. Electronics, 2022, 12(1). DOI: 10.3390/electronics12010029. [21] ANOOPA S, SALIM A.Survey on Anomaly Detection in Surveillance Videos. Materials Today(Proceedings), 2022, 58: 162-167. [22] CHANDRAKALA S, DEEPAK K, REVATHY G.Anomaly Detection in Surveillance Videos: A Thematic Taxonomy of Deep Models, Review and Performance Analysis. Artificial Intelligence Review, 2023, 56(4): 3319-3368. [23] DUONG H T, LE V T, HOANG V T.Deep Learning-Based Ano-maly Detection in Video Surveillance: A Survey. Sensors, 2023, 23(11). DOI: 10.3390/s23115024. [24] TRAN M T, VU N T, VO D N, et al. Anomaly Analysis in Images and Videos: A Comprehensive Review. ACM Computing Surveys, 2023, 55(7). DOI: 10.1145/3544014. [25] CAETANO F, CARVALHO P, CARDOSO J S.Unveiling the Performance of Video Anomaly Detection Models-A Benchmark-Based Review. Intelligent Systems with Applications, 2023. DOI: 10.1016/j.iswa.2023.200236. [26] 张晓平,纪佳慧,王力,等.基于视频的人体异常行为识别与检测方法综述.控制与决策, 2022, 37(1): 14-27. (ZHANG X P, JI J H, WANG L, et al. Overview of Video Based Human Abnormal Behavior Recognition and Detection Methods. Control and Decision, 2022, 37(1): 14-27.) [27] NAYAK R, PATI U C, DAS S K.A Comprehensive Review on Deep Learning-Based Methods for Video Anomaly Detection. Image and Vision Computing, 2021, 106. DOI: 10.1016/j.imavis.2020.104078. [28] GEORGESCU M I, IONESCU T R, KHAN S F, et al. A Background-Agnostic Framework with Adversarial Training for Abnormal Event Detection in Video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4505-4523. [29] POURREZA M, MOHAMMADI B, KHAKI M, et al. G2D: Generate to Detect Anomaly//Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2021: 2002-2011. [30] ZAHEER Z M, LEE H J, MAHMOOD A, et al. Stabilizing Adversarially Learned One-Class Novelty Detection Using Pseudo Anomalies. IEEE Transactions on Image Processing, 2022, 31: 5963-5975. [31] ZAHEER Z M, LEE J H, ASTRID M, et al. Old is Gold: Redefining the Adversarially Learned One-Class Classifier Training Paradigm//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 14171-14181. [32] ASTRID M, ZAHEER M Z, LEE S I. Synthetic Temporal Anomaly Guided End-to-End Video Anomaly Detection//Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 207-214. [33] QI X S, HU Z S, JI G L.Improved Video Anomaly Detection with Dual Generators and Channel Attention. Applied Sciences, 2023, 13(4). DOI: 10.3390/app13042284. [34] 郭方圆,吉根林.基于双鉴别器和伪视频生成的视频异常检测方法[J/OL].[2023-12-01]. https://link.cnki.net/urlid/50.1075.TP.20231130.1616.004. (GUO F Y, JI G L. Video Anomaly Detection Method Based on Dual Discriminators and Pseudo Video Generation[J/OL].[2023-12-01]. https://link.cnki.net/urlid/50.1075.TP.20231130.1616.004.) [35] XU K, SUN T F, JIANG X H.Video Anomaly Detection and Localization Based on an Adaptive Intra-Frame Classification Network. IEEE Transactions on Multimedia, 2020, 22(2): 394-406. [36] WU P, LIU J, LI M M, et al. Fast Sparse Coding Networks for Ano-maly Detection in Videos. Pattern Recognition, 2020, 107. DOI: 10.1016/j.patcog.2020.107515. [37] 肖进胜,郭浩文,谢红刚,等.监控视频异常行为检测的概率记忆自编码网络.软件学报, 2023, 34(9): 4362-4377. (XIAO J S, GUO H W, XIE H G, et al. Probabilistic Memory Auto-encoding Network for Abnormal Behavior Detection in Surveillance Videos. Journal of Software, 2023, 34(9): 4362-4377.) [38] HUANG C, YANG Z H, WEN J, et al. Self-Supervision-Augmented Deep Autoencoder for Unsupervised Visual Anomaly Detection. IEEE Transactions on Cybernetics, 2022, 52(12): 13834-13847. [39] 戚小莎,曾静,吉根林.双交叉注意力自编码器改进视频异常检测.南京师大学报(自然科学版), 2023, 46(1): 110-119. (QI X S, ZENG J, JI G L.Improved Video Anomaly Detection with Dual Criss-Cross Attention Auto Encoder. Journal of Nanjing Normal University(Natural Science Edition), 2023, 46(1): 110-119.) [40] 李欣璐,吉根林,赵斌.基于卷积自编码器分块学习的视频异常事件检测与定位.数据采集与处理, 2021, 36(3): 489-497. (LI X L, JI G L, ZHAO B.Convolutional Auto-Encoder Patch Learning Based Video Anomaly Event Detection and Localization. Journal of Data Acquisition and Processing, 2021, 36(3): 489-497.) [41] QI X S, JI G L, ZHANG J, et al. Multi Chunk Learning Based Auto Encoder for Video Anomaly Detection. Intelligent Automation and Soft Computing, 2022, 33(3): 1861-1875. [42] WANG J Q, ZHANG J, JI G L, et al. Criss-Cross Attention Based Auto Encoder for Video Anomaly Event Detection. Intelligent Automation and Soft Computing, 2022, 34(3): 1629-1642. [43] HAO Y, LI J, WANG N N, et al. Spatiotemporal Consistency-Enhanced Network for Video Anomaly Detection. Pattern Recognition, 2022, 121. DOI: 10.1016/j.patcog.2021.108232. [44] PILLAI G V, SEN D.Anomaly Detection in Nonstationary Videos Using Time-Recursive Differencing Network-Based Prediction. IEEE Geoscience and Remote Sensing Letters, 2022, 19. DOI: 10.1109/LGRS.2021.3072191. [45] 武光利,郭振洲,李雷霆,等.融合FCN和LSTM的视频异常事件检测.上海交通大学学报, 2021, 55(5): 607-614. (WU G L, GUO Z Z, LI L T, et al. Video Abnormal Detection Combining FCN with LSTM. Journal of Shanghai Jiaotong University, 2021, 55(5): 607-614.) [46] HUANG C, WEN J, XU Y, et al. Self-Supervised Attentive Generative Adversarial Networks for Video Anomaly Detection. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(11): 9389-9403. [47] HUANG C, WU Z H, WEN J, et al. Abnormal Event Detection Using Deep Contrastive Learning for Intelligent Video Surveillance System. IEEE Transactions on Industrial Informatics, 2022, 18(8): 5171-5179. [48] 孙奇,吉根林,张杰.基于非局部注意力生成对抗网络的视频异常事件检测方法.计算机科学, 2022, 49(8): 172-177. (SUN Q, JI G L, ZHANG J.Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection. Computer Science, 2022, 49(8): 172-177.) [49] 曾静,李莹,戚小莎,等.多层记忆增强生成对抗网络二次预测的视频异常检测方法.应用科学学报, 2023, 41(1): 80-94. (ZENG J, LI Y, QI X S, et al. Video Anomaly Detection Method Based on Secondary Prediction of Multi-layer Memory Enhancement Generative Adversarial Network. Journal of Applied Science-Electronics and Information Engineering, 2023, 41(1): 80-94.) [50] SULTANI W, CHEN C, SHAH M. Real-world Anomaly Detection in Surveillance Videos//Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 6479-6488. [51] WAN B Y, FANG Y M, XIA X, et al. Weakly Supervised Video Anomaly Detection via Center-Guided Discriminative Learning//Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2020. DOI: 10.1109/ICME46284.2020.9102722. [52] 肖进胜,申梦瑶,江明俊,等.融合包注意力机制的监控视频异常行为检测.自动化学报, 2020, 48(12): 2951-2959. (XIAO J S, SHEN M Y, JIANG M J, et al. Abnormal Behavior Detection Algorithm with Video-Bag Attention Mechanism in Surveillance Video. Acta Automatica Sinica, 2020, 48(12): 2951-2959.) [53] SHARIF M H, JIAO L, OMLIN W C.CNN-ViT Supported Weakly-Supervised Video Segment Level Anomaly Detection. Sensors, 2023, 23(18). DOI: 10.3390/s23187734. [54] LI C, CHEN M.Dy-MIL: Dynamic Multiple-Instance Learning Frame-work for Video Anomaly Detection. Multimedia Systems, 2024, 30(1). DOI: 10.21203/rs.3.rs-2906577/v1. [55] 魏思倩,吉根林,许振,等.利用注意力机制的多示例学习视频异常检测.小型微型计算机系统, 2022, 43(12): 2575-2579. (WEI S Q, JI G L, XU Z, et al. Attention Mechanism Based Multiple Instance Learning Video Anomaly Detection. Journal of Chinese Computer Systems, 2022, 43(12): 2575-2579.) [56] 黄敏,尚瑞欣,钱惠敏.面向视频中人体行为识别的复合型深度神经网络.模式识别与人工智能, 2022, 35(6): 562-570. (HUANG M, SHANG R X, QIAN H M.Composite Deep Neural Network for Human Activities Recognition in Video. Pattern Re-cognition and Artificial Intelligence, 2022, 35(6): 562-570.) [57] MAHADEVAN V, LI W X, BHALODIA V, et al. Anomaly Detection in Crowded Scenes//Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 1975-1981. [58] LU C W, SHI J P, JIA J Y. Abnormal Event Detection at 150 FPS in MATLAB//Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2013: 2720-2727. [59] LUO W X, LIU W, GAO S H. A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework//Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 341-349.
2024, Vol. 37 
