Vision-Based Abnormal Vehicle Behavior Detection: A Survey
HUANG Chao1,3, HU Zhijun2, XU Yong1,3, WANG Yaowei3
1.Bio-Computing Research Center, College of Computer Science and Technology, Harbin Institute of Technology(Shenzhen), Shenzhen 518055; 2.College of Mathematics and Statistics, Guangxi Normal University, Guilin 541004; 3.Peng Cheng Laboratory, Shenzhen 518000
Abstract：Vision-based abnormal vehicle behavior detection can detect abnormal vehicle behaviors in the traffic surveillance video promptly and give an alarm. It plays an important role in improving the efficiency of traffic enforcement and traffic conditions and reducing traffic accident rate. Despite the progresses in abnormal vehicle behavior detection, there are still many challenges in practical application, such as lack of labeled data, uncertain anomaly, occlusion and poor real time capability. To make a clear understanding of abnormal vehicle behavior detection, the algorithms proposed in recent years are summarized. Firstly, the typical features representing vehicle behaviors are introduced, and the advantages and disadvantages of model learning methods of the algorithms are discussed from the perspectives of supervised and unsupervised learning. Then, the existing algorithms are categorized into model-based, reconstruction-based and deep neural network-based methods. Each category is introduced and analyzed. Finally, problems and prediction of the future of abnormal vehicle behavior detection are discussed.
[1] JEONG H, YOO Y, YI K M, et al. Two-Stage Online Inference Model for Traffic Pattern Analysis and Anomaly Detection. Machine Vision and Applications, 2014, 25(6): 1501-1517. [2] YUAN Y, FANG J W, WANG Q. Online Anomaly Detection in Crowd Scenes via Structure Analysis. IEEE Transactions on Cybernetics, 2015, 45(3): 562-575. [3] YUAN Y, WANG D, WANG Q. Anomaly Detection in Traffic Scenes via Spatial-Aware Motion Reconstruction. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(5): 1198-1209. [4] PATHAK D, SHARANG A, MUKERJEE A. Anomaly Localization in Topic-Based Analysis of Surveillance Videos // Proc of the IEEE Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2015: 389-395. [5] SABOKROU M, FAYYAZ M, FATHY M, et al. Deep-Cascade: Cascading 3D Deep Neural Networks for Fast Anomaly Detection and Localization in Crowded Scenes. IEEE Transactions on Image Processing, 2017, 26(4): 1992-2004. [6] HU X, HU S Q, HUANG Y P, et al. Video Anomaly Detection Using Deep Incremental Slow Feature Analysis Network. IET Computer Vision, 2016, 10(4): 258-265. [7] CHENG K W, CHEN Y T, FANG W H. Gaussian Process Regre-ssion-Based Video Anomaly Detection and Localization with Hierarchical Feature Representation. IEEE Transactions on Image Proce-ssing, 2015, 24(12): 5288- 5301. [8] KALTSA V, BRIASSOULI A, KOMPATSIARIS I, et al. Multiple Hierarchical Dirichlet Processes for Anomaly Detection in Traffic. Computer Vision and Image Understanding, 2018, 169: 28-39. [9] TIAN B, YAO Q M, GU Y, et al. Video Processing Techniques for Traffic Flow Monitoring: A Survey // Proc of the IEEE International Conference on Intelligent Transportation Systems. Washington, USA: IEEE, 2011: 1103-1108. [10] SODEMANN A A, ROSS M P, BORGHETTI B J. A Review of Anomaly Detection in Automated Surveillance. IEEE Transactions on Systems, Man, and Cybernetics(Applications and Reviews), 2012, 42(6): 1257-1272. [11] SIVARAMAN S, TRIVEDI M M. Looking at Vehicles on the Road: A Survey of Vision-Based Vehicle Detection, Tracking, and Behavior Analysis. IEEE Transactions on Intelligent Transportation Systems, 2013, 14(4): 1773-1795. [12] TIAN B, MORRIS B T, TANG M, et al. Hierarchical and Networked Vehicle Surveillance in ITSC: A Survey. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(2): 557-580. [13] DATONDJI S R E, DUPUIS Y, SUBIRATS P, et al. A Survey of Vision-Based Traffic Monitoring of Road Intersections. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(10): 2681-2698. [14] SHIRAZI M S, MORRIS B T. Looking at Intersections: A Survey of Intersection Monitoring, Behavior and Safety Analysis of Recent Studies. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(1): 4-24. [15] YUN K M, JEONG H, YI K M, et al. Motion Interaction Field for Accident Detection in Traffic Surveillance Video // Proc of the 22nd International Conference on Pattern Recognition. Washington, USA: IEEE, 2014: 3062-3067. [16] EMONET R, VARADARAHAN J, ODOBEZ J M. Temporal Ana-lysis of Motif Mixtures Using Dirichlet Processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(1): 140-156. [17] ANJANA D K M, MURTHY O V R. Optical Flow Based Anomaly Detection in Traffic Scenes // Proc of the IEEE International Conference on Computational Intelligence and Computing Research. Washington, USA: IEEE, 2017: 1-7. [18] SALEEMI I, HARTUNG L, SHAH M. Scene Understanding by Statistical Modeling of Motion Patterns // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2010: 2069-2076. [19] SABOKROU M, FATHY M, HOSEINI M, et al. Real-Time Ano-maly Detection and Localization in Crowded Scenes // Proc of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2015: 56-62. [20] COLQUE R V H M, CAETANO C, DE ANDRADE M T L, et al. Histograms of Optical Flow Orientation and Magnitude and Entropy to Detect Anomalous Events in Videos. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27(3): 673-682. [21] HASAN M, CHOI J, NEUMANN J, et al. Learning Temporal Re-gularity in Video Sequences // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 733-742. [22] KAVIANI R, AHMADI P, GHOLAMPOUR I. Incorporating Fully Sparse Topic Models for Abnormality Detection in Traffic Videos // Proc of the 4th International Conference on Computer and Knowledge Engineering. Washington, USA: IEEE, 2014: 586-591. [23] LI H, ZHANG Y H, YANG M, et al. A Rapid Abnormal Event Detection Method for Surveillance Video Based on a Novel Feature in Compressed Domain of HEVC // Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2014. DOI: 10.1109/ICME.2014.6890212. [24] LI Y S, LIU W M, HUANG Q H. Traffic Anomaly Detection Based on Image Descriptor in Videos. Multimedia Tools and Applications, 2016, 75(5): 2487-2505. [25] NALLAIVAROTHAYAN H, FOOKES C, DENMAN S, et al. An MRF Based Abnormal Event Detection Approach Using Motion and Appearance Features // Proc of the 11th IEEE International Conference on Advanced Video and Signal Based Surveillance. Washington, USA: IEEE, 2014: 343-348. [26] RASHEED N, KHAN S A, KHALID A. Tracking and Abnormal Behavior Detection in Video Surveillance Using Optical Flow and Neural Networks // Proc of the 28th International Conference on Advanced Information Networking and Applications Workshops. Washington, USA: IEEE, 2014: 61-66. [27] ZHONG Z F, ZHANG B, LU G M, et al. An Adaptive Background Modeling Method for Foreground Segmentation. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(5): 1109-1121. [28] WEN J J, XU Y, TANG J H, et al. Joint Video Frame Set Division and Low-Rank Decomposition for Background Subtraction. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(12): 2034-2048. [29] LIRA G, KOKKINOGENIS Z, ROSSETTI R J F, et al. A Computer-Vision Approach to Traffic Analysis over Intersections // Proc of the IEEE International Conference on Intelligent Transportation Systems. Washington, USA: IEEE, 2016: 47-53. [30] ZHONG Z F, WEN J J, ZHANG B, et al. A General Moving Detection Method Using Dual-Target Nonparametric Background Mo-del. Knowledge-Based Systems, 2019, 164: 85-95. [31] ULLAH H, ULLAH M, AFRIDI H, et al. Traffic Accident Detection through a Hydrodynamic Lens // Proc of the IEEE International Conference on Image Processing. Washington, USA: IEEE, 2015: 2470-2474. [32] ZHONG Z F, XU Y, LI Z Y, et al. Background Modelling Using Discriminative Motion Representation. IET Computer Vision, 2017, 11(6): 463-470. [33] WEI J Y, ZHAO J F, ZHAO Y Y, et al. Unsupervised Anomaly Detection for Traffic Surveillance Based on Background Modeling // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 1292-1297. [34] SANTHOSH K K, DOGRA D P, ROY P P, et al. Trajectory-Based Scene Understanding Using Dirichlet Process Mixture Model. IEEE Transactions on Cybernetics, 2019. DOI: 10.1109/TCYB.2019.2931139. [35] LIN W Y, ZHOU Y, XU H T, et al. A Tube-and-Droplet-Based Approach for Representing and Analyzing Motion Trajectories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(8): 1489-1503. [36] SANTHOSH K K, DOGRA D P, ROY P P, et al. Video Trajectory Classification and Anomaly Detection Using Hybrid CNN-VAE [J/OL]. [2019-08-18]. https://arxiv.org/pdf/1812.07203.pdf. [37] MOU L T, MAO S S, XIE H T, et al. Structured Behaviour Prediction of On-Road Vehicles via Deep Forest. Electronics Letters, 2019, 55(8): 452-455. [38] TANG X F, GONG B Q, YU Y W, et al. Joint Modeling of Dense and Incomplete Trajectories for Citywide Traffic Volume Inference [C/OL]. [2019-08-18]. https://arxiv.org/pdf/1902.09255v1.pdf. [39] GIANNAKERIS P, KALTSA V, AVGERINAKIS K, et al. Speed Estimation and Abnormality Detection from Surveillance Cameras // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 93-99. [40] VALLEJO D, ALBUSAC J, JIMENEZ L, et al. A Cognitive Surveillance System for Detecting Incorrect Traffic Behaviors. Expert Systems with Applications, 2009, 36(7): 10503-10511. [41] ALBUSAC J, VALLEJO D, JIMENEZ-LINARES L, et al. Intelligent Survei-llance Based on Normality Analysis to Detect Abnormal Behaviors. International Journal of Pattern Recognition and Artificial Intelligence, 2009, 23(7): 1223-1244. [42] SAFARA F, EFTEKHARI-MOGHADAM A. Knowledge Discovery of Traffic/People Behaviors Based on Image Mining Approach // Proc of the Geometric Modeling and Imaging-New Trends. Wa-shington, USA: IEEE, 2006: 255-258. [43] BISWAS S, BABU R V. Anomaly Detection via Short Local Tra-jectories. Neurocomputing, 2017, 246: 63-72. [44] JODOIN P, KONRAD J, SALIGRAMA V. Modeling Background Activity for Behavior Subtraction // Proc of the 2nd ACM/IEEE International Conference on Distributed Smart Cameras. Washington, USA: IEEE, 2008. DOI: 10.1109/ICDSC.2008.4635683. [45] DONG N, JIA Z, SHAO J, et al. Traffic Abnormality Detection through Directional Motion Behavior Map // Proc of the 7th IEEE International Conference on Advanced Video and Signal Based Surveillance. Washington, USA: IEEE, 2010: 80-84. [46] BENEZETH Y, JODOIN P M, SALIGRAMA V, et al. Abnormal Events Detection Based on Spatio-Temporal Co-occurences // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2009: 2458-2465. [47] SULTANI W, CHEN C, SHAH M. Real-World Anomaly Detection in Surveillance Videos // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 6479-6488. [48] CHEN Y, YU Y L, LI T. A Vision Based Traffic Accident Detection Method Using Extreme Learning Machine // Proc of the International Conference on Advanced Robotics and Mechatronics. Washington, USA: IEEE, 2016: 567-572. [49] HINAMI R, MEI T, SATOH S. Joint Detection and Recounting of Abnormal Events by Learning Deep Generic Knowledge // Proc of the IEEE International Conference on Computer Vision. Wa-shington, USA: IEEE, 2017: 3639-3647. [50] KAMIJO S, MATSUSHITA Y, IKEUCHI K, et al. Traffic Monitoring and Accident Detection at Intersections. IEEE Transactions on Intelligent Transportation Systems, 2000, 1(2): 108-118. [51] CAI Y F, WANG H, CHEN X B, et al. Trajectory-Based Anomalous Behaviour Detection for Intelligent Traffic Surveillance. IET Intelligent Transport Systems, 2015, 9(8): 810-816. [52] FAROOQ M U, KHAN N A, ALI M S, et al. Unsupervised Video Surveillance for Anomaly Detection of Street Traffic. International Journal of Advanced Computer Science and Applications, 2017, 8(12): 270-275. [53] GHRAB N B, FENDRI E, HAMMAMI M. Abnormal Events Detection Based on Trajectory Clustering // Proc of the 13th International Conference on Computer Graphics, Imaging and Visualization. Washington, USA: IEEE, 2016: 301-306. [54] GOSHORN R, GOSHORN D, GOSHORN J, et al. Abnormal Behavior-Detection Using Sequential Syntactical Classification in a Network of Clustered Cameras // Proc of the 2nd ACM/IEEE International Conference on Distributed Smart Cameras. Washington, USA: IEEE, 2008. DOI: 10.1109/ICDSC.2008.4635732. [55] NGAN H Y T, YUNG N H C, YEH A G O. Outlier Detection in Traffic Data Based on the Dirichlet Process Mixture Model. IET Intelligent Transport Systems, 2015, 9(7): 773-781. [56] RANJITH R, ATHANESIOUS J J, VAIDEHI V. Anomaly Detection Using DBSCAN Clustering Technique for Traffic Video Surveillance // Proc of the 7th International Conference on Advanced Computing. Washington, USA: IEEE, 2015. DOI: 10.1109/ICoAC.2015.7562795. [57] RIBEIRO M, LAZZARETTI A E, LOPES H S. A Study of Deep Convolutional Auto-Encoders for Anomaly Detection in Videos. Pattern Recognition Letters, 2018, 105: 13-22. [58] LIU W, LUO W X, LIAN D Z, et al. Future Frame Prediction for Anomaly Detection-A New Baseline[C/OL]. [2019-08-25]. https://arxiv.org/pdf/1712.09867.pdf. [59] ROSTHTKHARI M J, LEVINE M D. Online Dominant and Ano-malous Behavior Detection in Videos // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2013: 2611-2618. [60] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5967-5976. [61] WEN J J, LAI Z H, MING Z, et al. Directional Gaussian Model for Automatic Speeding Event Detection. IEEE Transactions on Information Forensics and Security, 2017, 12(10): 2292-2307. [62] ZU H, XIE Y H, MA L, et al. Vision-Based Real-Time Traffic Accident Detection // Proc of the 11th World Congress on Intelligent Control and Automation. Washington, USA: IEEE, 2014: 1035-1038. [63] AKÖZ Ö, KARSLIGIL M E. Traffic Event Classification at Intersections Based on the Severity of Abnormality. Machine Vision and Applications, 2014, 25: 613-632. [64] LOY C C, XIANG T, GONG S G. Detecting and Discriminating Behavioural Anomalies. Pattern Recognition, 2011, 44(1): 117-132. [65] KIM J, GRAUMAN K. Observe Locally, Infer Globally: A Space-Time MRF for Detecting Abnormal Activities with Incremental Updates // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2009: 2921-2928. [66] KUMARAN S K, DOGRA D P, ROY P P. Queuing Theory Guid-ed Intelligent Traffic Scheduling through Video Analysis Using Dirichlet Process Mixture Model. Expert Systems with Applications, 2019, 118: 169-181. [67] SANTHOSH K K, DOGRA D P, ROY P. Temporal Unknown Incremental Clustering Model for Analysis of Traffic Surveillance Vi-deos. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(5): 1762-1773. [68] CONG Y, YUAN J S, LIU J. Sparse Reconstruction Cost for Abnormal Event Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2011: 3449-3456. [69] TIAN C W, XU Y, FEI L K, et al. Enhanced CNN for Image Denoising. CAAI Transactions on Intelligence Technology, 2019, 4(1): 17-23. [70] TIAN C W, XU Y, ZUO W M. Image Denoising Using Deep CNN with Batch Renormalization[C/OL]. [2019-08-25]. https://doi.org/10.1016/j.neunet.2019.08.022. [71] XU D, YAN Y, RICCI E, et al. Detecting Anomalous Events in Videos by Learning Deep Representations of Appearance and Mo-tion. Computer Vision and Image Understanding, 2017, 156: 117-127. [72] ZHAO Y R, DENG B, SHEN C, et al. Spatio-Temporal Autoencoder for Video Anomaly Detection // Proc of the 25th ACM International Conference on Multimedia. New York, USA: ACM, 2017: 1933-1941. [73] MEDEL J R, SAVAKIS A. Anomaly Detection in Video Using Predictive Convolutional Long Short-Term Memory Networks[C/OL]. [2019-08-15]. https://arxiv.org/ftp/arxiv/papers/1612/1612.00390.pdf. [74] SABOKROU M, KHALOOEI M, FATHY M, et al. Adversarially Learned One-Class Classifier for Novelty Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 3379-3388. [75] ZHAO B, LI F F, XING E P. Online Detection of Unusual Events in Videos via Dynamic Sparse Coding // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2011: 3313-3320. [76] LI C, HAN Z J, YE Q X, et al. Visual Abnormal Behavior Detection Based on Trajectory Sparse Reconstruction Analysis. Neurocomputing, 2013, 119: 94-100. [77] CHEN Z J, WU C Z, ZHANG Y S, et al. Vehicle Behavior Lear-ning via Sparse Reconstruction with l2-lp Minimization and Trajectory Similarity. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(2): 236-247. [78] LU C W, SHI J P, JIA J Y. Abnormal Event Detection at 150 FPS in Matlab // Proc of the IEEE International Conference on Compu-ter Vision. Washington, USA: IEEE, 2013: 2720-2727. [79] YU B S, LIU Y Z, SUN Q S. A Content-Adaptively Sparse Reconstruction Method for Abnormal Events Detection with Low-Rank Property. IEEE Transactions on Systems, Man, and Cybernetics(Systems), 2017, 47(4): 704-716. [80] SABOKROU M, FATHY M, HOSEINI M. Video Anomaly Detection and Localisation Based on the Sparsity and Reconstruction Error of Auto-Encoder. Electronics Letters, 2016, 52(13): 1122-1124. [81] LUO W X, LIU W, GAO S H. A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 341-349. [82] SINGH D, MOHAN C K. Deep Spatio-Temporal Representation for Detection of Road Accidents Using Stacked Autoencoder. IEEE Transactions on Intelligent Transportation Systems, 2019, 20(3): 879-887. [83] CHANG M C, WEI Y, SONG N H, et al. Video Analytics in Smart Transportation for the AIC′18 Challenge // Proc of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2018: 61-68.
2020, Vol. 33 
