Rain Removal Method for Traffic Surveillance Video in Joint Spatial-Frequency Domain
SONG Chuanming1,3, HONG Xu1,2, LIU Dingkun1, WANG Xianghai1
1. School of Computer and Information Technology, Liaoning Nor-mal University, Dalian 116081 2. School of Information Engineering, Liaoning Institute of Science and Engineering, Jinzhou 121000 3. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023
Abstract The processing of traffic surveillance videos in rainy days is inefficient and unreliable. A rain removal method is proposed for traffic surveillance video in joint spatial-frequency domain with an ability to discriminate the amount of rainfall adaptively. After setting the low frequency coefficients of non-subsampled Shearlet transform to zero, the map of all edge information is computed by the Otsu method. Subsequently, the saliency mapping method is utilized to calculate a depth map. The map of main edge information is obtained by bilateral filtering and the high-frequency coefficients are retained. Through combining the map of all edge information, the map of main edge information and the frame difference, the areas of both raindrop and rain line are determined, and the amount of rainfall is analyzed. If the rain is moderate or heavy, a curvature-driven diffusion method is employed to restore the pixels in the areas of raindrop and rain line. Otherwise, the detection results are aggregated under two scales. Experimental results show that the proposed algorithm effectively removes the raindrops and rain lines in videos with shapes and texture details of the objects preserved. Moreover, the post-processing quality is improved, such as moving object tracking.
Fund:Natural Science Foundation of Liaoning Province(No.20180550570), Program for Liaoning Excellent Talents in University(No.[2018] 478, 64th), Open Foundation of State Key Laboratory for Novel Software Technology of Nanjing University(No.KFKT2018B07)
Corresponding Authors:
SONG Chuanming, Ph.D., professor. His research interests include image and video coding, and traffic surveillance video processing.
About author:: HONG Xu, master, teaching assistant. His research interests include traffic surveillance video processing.LIU Dingkun, master student. Her research interests include screen content video coding and video information processing.WANG Xianghai, Ph.D., professor. His research interests include multimedia information processing, computer graphics and remote sensing information processing.
[1] 刘巧元,王玉茹,张金玲,等.基于相关滤波器的视频跟踪方法研究进展.自动化学报, 2019, 45(2): 265-275. (LIU Q Y, WANG Y R, ZHANG J L, et al. Research Progress of Visual Tracking Methods Based on Correlation Filter. Acta Automa-tica Sinica, 2019, 45(2): 265-275.) [2] 李 力,王飞跃.地面交通控制的百年回顾和未来展望.自动化学报, 2018, 44(4): 577-583. (LI L, WANG F Y. Ground Traffic Control in the Past Century and Its Future Perspective. Acta Automatica Sinica, 2018, 44(4): 577-583.) [3] 李鹏越,田建东,王国霖,等.面向机器人环境共融的图像去雪算法.机械工程学报, 2019, 55(11): 98-104. (LI P Y, TIAN J D, WANG G L, et al. Image Snow Removal Methods for Robotic Environment Fusion. Journal of Mechanical Engineering, 2019, 55(11): 98-104.) [4] LI P Y, YUN M S, TIAN J D, et al. Stacked Dense Networks for Single-Image Snow Removal. Neurocomputing, 2019, 367: 152-163. [5] 周浦城,周 远,韩裕生.视频图像去雨技术研究进展.图学学报, 2017, 38(5): 629-646. (ZHOU P C, ZHOU Y, HAN Y S. Review of Rain Removal Techniques in Videos and Images. Journal of Graphics, 2017, 38(5): 629-646.) [6] XUE X W, JIN X, ZHANG C Y, et al. Motion Robust Rain Detection and Removal from Videos // Proc of the 14th IEEE Internatio-nal Workshop on Multimedia Signal Processing. Washington, USA: IEEE, 2012: 170-174. [7] GARG K, NAYAR S K. Detection and Removal of Rain from Videos // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2004. DOI: 10.1109/CVPR.2004.1315077. [8] GARG K, NAYAR S K. Photorealistic Rendering of Rain Streaks. ACM Transactions on Graphics, 2006, 25(3): 996-1002. [9] GARG K, NAYAR S K. Vision and Rain. International Journal of Computer Vision, 2007, 75(1): 3-27. [10] 张颖翔,陈 强,刘允才.视频图像中雨滴检测与去除方法研究.微型电脑应用, 2007, 23(12): 16-20. (ZHANG Y X, CHEN Q, LIU Y C. The Rain Drops Are Mea-sured and Get Rid of the Method to Study in the Video Picture. Microcomputer Applications, 2007, 23(12): 16-20.) [11] TRIPATHI A K, MUKHOPADHYAY S. A Probabilistic Approach for Detection and Removal of Rain From Videos. IETE Journal of Research, 2011, 57(1): 82-91. [12] TRIPATHI A K, MUKHOPADHYAY S. Video Post Processing: Low-Latency Spatiotemporal Approach for Detection and Removal of Rain. IET Image Processing, 2012, 6(2): 181-196. [13] PARK W J, LEE K H. Rain Removal Using Kalman Filter in Video // Proc of the International Conference on Smart Manufacturing Application. Washington, USA: IEEE, 2008: 494-497. [14] BARNUM P C, NARASIMHAN S, KANADE T. Analysis of Rain and Snow in Frequency Space. International Journal of Computer Vision, 2010, 86(2/3): 256-274. [15] BREWER N, LIU N J. Using the Shape Characteristics of Rain to Identify and Remove Rain from Video // Proc of the Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition and Structural and Syntactic Pattern Recognition. Berlin, Germany: Springer, 2008: 451-458. [16] KANG L W, LIN C W, FU Y H. Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition. IEEE Transaction on Image Process, 2012, 21(4): 1742-1755. [17] HUANG D A, KANG L W, YANG M C, et al. Context-Aware Single Image Rain Removal // Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2012: 164-169. [18] 肖进胜,王 文,邹文涛,等.基于景深和稀疏编码的图像去雨算法.计算机学报, 2019, 42(9): 2024-2034. (XIAO J S, WANG W, ZOU W T, et al. An Image Rain Removal Algorithm via Depth of Field and Sparse Coding. Chinese Journal of Computers, 2019, 42(9): 2024-2034.) [19] 汤红忠,王 翔,张小刚,等.面向单幅图像去雨的非相干字典学习及其稀疏表示研究.通信学报, 2017, 38(7): 28-35. (TANG H Z, WANG X, ZHANG X G, et al. Incoherent Dictio-nary Learning and Sparse Representation for Single-Image Rain Removal. Journal on Communications, 2017, 38(7): 28-35.) [20] 汤红忠,刘 婷,曾淑英,等.群稀疏残差约束的引导字典学习算法及其单幅图像去雨[J/OL]. [2020-07-20]. http://kns.cnki.net/kcms/detail/11.2925.TP.20200120.1005.002.html. (TANG H Z, LIU T, ZENG S Y, et al. Guided Dictionary Learning Algorithm with Group Sparse Residual Constraints for Single Image Deraining [J/OL]. [2020-07-20]. http://kns.cnki.net/kcms/detail/11.2925.TP.20200120.1005.002.html.) [21] WANG Y T, ZHAO X L, JIANG T X, et al. A Total Variation and Group Sparsity Based Tensor Optimization Model for Video Rain Streak Removal. Signal Processing: Image Communication, 2019, 73(1): 96-108. [22] 林向伟,曾焕强,侯进辉,等.基于多细节卷积神经网络的单幅图像去雨方法.信号处理, 2019, 35(3): 460-465. (LIN X W, ZENG H Q, HOU J H, et al. Multi-detail Convolutional Neural Networks for Single Image Rain Removal. Journal of Signal Processing, 2019, 35(3): 460-465.) [23] JIN X, CHEN Z B, LI W P. AI-GAN: Asynchronous Interactive Generative Adversarial Network for Single Image Rain Removal. Pattern Recognition, 2020, 100. DOI: 10.1016/j.patcog.2019.107143. [24] LI S Y, REN W Q, ZHANG J W, et al. Single Image Rain Removal via a Deep Decomposition-Composition Network. Computer Vision and Image Understanding, 2019, 186: 48-57. [25] SHI Z S, LI Y W, ZHANG C Q, et al. Weighted Median Guided Filtering Method for Single Image Rain Removal[C/OL]. [2020-07-20]. https://doi.org/10.1186/s13640-018-0275-9. [26] LI M H, XIE Q, ZHAO Q, et al. Video Rain Streak Removal by Multiscale Convolutional Sparse Coding // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 6644-6653. [27] TOMASI C, MANDUCHI R. Bilateral Filtering for Gray and Color Images // Proc of the 6th International Conference on Computer Vision. Washington, USA: IEEE, 1998: 839-846. [28] KIM J H, SIM J Y, KIM C S. Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion. IEEE Transactions on Image Processing, 2015, 24(9): 2658-2670. [29] WEI W, YI L X, XIE Q, et al. Should We Encode Rain Streaks in Video as Deterministic or Stochastic? // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2535-2544. [30] REN W H, TIAN J D, HAN Z, et al. Video Desnowing and De-raining Based on Matrix Decomposition // Proc of the IEEE International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2838-2847. [31] JIANG T X, HUANG T Z, ZHAO X L, et al. A Novel Tensor-Based Video Rain Streaks Removal Approach via Utilizing Discriminatively Intrinsic Priors // Proc of the IEEE International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2818-2827.
2020, Vol. 33 
