Multi-granularity Dynamic Scene Image Deblurring Network Based on Deep Fusion of Frequency Domain and Spatial Domain Features
CHEN Zihan1,2, ZHANG Hongyun1,2, MIAO Duoqian1,2, CAI Kecan1,2
1. College of Electronic and Information Engineering, Tongji University, Shanghai 201804; 2. Key Laboratory of Embedded Systems and Service Computing of Ministry of Education, Tongji University, Shanghai 201804
Abstract Dynamic scene image deblurring is highly ill-posed, and the relative motion between the camera and the photographed target makes the blur non-uniform. Most existing deep learning methods focus on the spatial domain processing and neglect the potential contribution of the frequency domain to structural and detail recovery, leading to poor deblurring results. To solve the problems, the role of frequency domain information in image deblurring is rethought, and multi-granularity dynamic scene image deblurring network based on deep fusion of frequency domain and spatial domain features is proposed. Firstly, a frequency domain gated frequency-spatial feature deep fusion module is proposed to fully explore the correlation between spatial domain and frequency domain information. The redundancy of the fused features is reduced and the complementarity between the two domains is enhanced. Secondly, based on the proposed module, a multi-granularity network is constructed, and it fully utilizes different granularity information in the spatial domain and frequency domain for coarse-to-fine image deblurring. Finally, to solve the problem of frequency domain feature map resolution mismatch caused by different input feature map sizes during training and testing, a frequency domain resolution adaptive testing strategy is adopted to maintain the consistency of frequency changes. Experiments conducted on synthetic datasets, GoPro and HIDE, and a real dataset, RealBlur, demonstrate the proposed algorithm outperforms existing advanced algorithms in reconstructing clear images with competitive parameters and efficiency.
Fund:National Key Research and Development Program of China(No.2022YFB3104700), National Natural Science Foun-dation of China(No.62076182,62376198), Natural Science Foun?dation of Shanghai(No.22ZR1466700)
Corresponding Authors:
ZHANG Hongyun, Ph.D., associate professor. Her research interests include principal curve algorithm, gra-nular computing and computer vision.
About author:: CHEN Zihan, Master student. Her research interests include computer vision and granular computing. MIAO Duoqian, Ph.D., professor. His research interests include machine learning, data mining, granular computing, artificial intelligence and text image processing.CAI Kecan, Ph.D. candidate. Her research interests include computer vision and granular computing.
CHEN Zihan,ZHANG Hongyun,MIAO Duoqian等. Multi-granularity Dynamic Scene Image Deblurring Network Based on Deep Fusion of Frequency Domain and Spatial Domain Features[J]. Pattern Recognition and Artificial Intelligence, 2024, 37(6): 557-569.
[1] WANG W, ZHANG J, ZHAI W, et al. Robust Object Detection via Adversarial Novel Style Exploration. IEEE Transactions on Image Processing, 2022, 31: 1949-1962. [2] FRANKE U, JOOS A. Real-Time Stereo Vision for Urban Traffic Scene Understanding // Proc of the IEEE Intelligent Vehicles Symposium. Washington, USA: IEEE, 2000: 273-278. [3] ZHANG K H, REN W Q, LUO W H, et al. Deep Image Deblu-rring: A Survey. International Journal of Computer Vision, 2022, 130(9): 2103-2130. [4] FERGUS R, SINGH B, HERTZMANN A, et al. Removing Camera Shake from a Single Photograph. ACM Transactions on Graphics, 2006, 25(3): 787-794. [5] CAI J F, JI H, LIU C Q, et al. Blind Motion Deblurring from a Single Image Using Sparse Approximation // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2009: 104-111. [6] CHO S, LEE S. Fast Motion Deblurring. ACM Transactions on Gra-phics, 2009, 28(5). DOI: 10.1145/1618452.161849. [7] PAN J S, SUN D Q, PFISTER H, et al. Blind Image Deblurring Using Dark Channel Prior // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1628-1636. [8] NAH S, KIM H T, LEE K M. Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 257-265. [9] TAO X, GAO H Y, SHEN X Y, et al. Scale-Recurrent Network for Deep Image Deblurring // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 8174-8182. [10] KUPYN O, MARTYNIUK T, WU J R, et al. DeblurGAN-v2: Deblurring(Orders-of-Magnitude) Faster and Better // Proc of the IEEE/CVF International Conference on Computer Vision. Wa-shington, USA: IEEE, 2019: 8877-8886. [11] ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage Progressive Image Restoration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 14816-14826. [12] CHO S J, JI S W, HONG J P, et al. Rethinking Coarse-to-Fine Approach in Single Image Deblurring // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 4621-4630. [13] ZAMIR S W, ARORA A, KHAN S, et al. Restormer: Efficient Transformer for High-Resolution Image Restoration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 5718-5729. [14] HAI J, YANG R, YU Y Q,et al. Combining Spatial and Frequency Information for Image Deblurring. IEEE Signal Processing Le-tters, 2022, 29: 1679-1683. [15] KONG L S, DONG J X, GE J J, et al. Efficient Frequency Domain-Based Transformers for High-Quality Image Deblurring // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 5886-5895. [16] CUI Y N, TAO Y, BING Z S, et al. Selective Frequency Network for Image Restoration[C/OL].[2024-04-27]. https://openreview.net/pdf?id=tyZ1ChGZIKO. [17] MAO X T, LIU Y M, LIU F Z, et al. Intriguing Findings of Frequency Selection for Image Deblurring. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(2): 1905-1913. [18] ZHANG B, SUN J, SUN F M, et al. Image Deblurring Method Based on Self-Attention and Residual Wavelet Transform. Expert Systems with Applications, 2024, 244. DOI: 10.1016/j.eswa.2023.123005. [19] 石林波,李华锋,张亚飞,等. 模态不变性特征学习和一致性细粒度信息挖掘的跨模态行人重识别. 模式识别与人工智能, 2022, 35(12): 1064-1077. (SHI L B, LI H F, ZHANG Y F, et al. Modal Invariance Feature Learning and Consistent Fine-Grained Information Mining Based Cross-Modal Person Re-identification. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1064-1077.) [20] 韩雪昆,苗夺谦,张红云,等. 双分支多粒度局部对齐的实例级草图图像检索. 模式识别与人工智能, 2023, 36(8): 701-711. (HAN X K, MIAO D Q, ZHANG H Y, et al. Instance-Level Sketch-Based Image Retrieval Based on Two Stream Multi-granula-rity Local Alignment Network. Pattern Recognition and Artificial Intelligence, 2023, 36(8): 701-711.) [21] ZHUGE Y Z, JIA X. Multi-granularity Transformer for Image Super-Resolution // Proc of the Asian Conference on Computer Vision. Berlin, Germany: Springer, 2023: 138-154. [22] 周天奕,丁卫平,黄嘉爽,等. 模糊逻辑引导的多粒度深度神经网络. 模式识别与人工智能, 2023, 36(9): 778-792. (ZHOU T Y, DING W P, HUANG J S, et al. Fuzzy Logic Guided Deep Neural Network with Multi-granularity. Pattern Recognition and Artificial Intelligence, 2023, 36(9): 778-792.) [23] ZOU W B, JIANG M C, ZHANG Y C, et al. SDWNet: A Straight Dilated Network with Wavelet Transformation for Image Deblurring // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 1895-1904. [24] CHANG H, YEUNG D Y, XIONG Y M. Super-Resolution Through Neighbor Embedding // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2004. DOI: 10.1109/CVPR.2004.1315043. [25] CHU X J, CHEN L Y, CHEN C P, et al. Improving Image Restoration by Revisiting Global Information Aggregation // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 53-71. [26] GUO S, YONG H W, ZHANG X D, et al. Spatial-Frequency Atten-tion for Image Denoising[C/OL].[2024-04-27]. https://arxiv.org/pdf/2302.13598.pdf. [27] SHEN Z Y, WANG W G, LU X K, et al. Human-Aware Motion Deblurring // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 5571-5580. [28] RIM J, LEE H, WON J, et al. Real-World Blur Dataset for Lear-ning and Benchmarking Deblurring Algorithms // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 184-201. [29] SHAO M W, BAO Z Y, LIU W H, et al. Frequency Domain-Enhanced Transformer for Single Image Deraining. The Visual Computer, 2024. DOI: 10.1007/s00371-023-03252-8. [30] TSAI F J, PENG Y T, TSAI C C, et al. BANet: A Blur-Aware Attention Network for Dynamic Scene Deblurring. IEEE Transactions on Image Processing, 2022, 31: 6789-6799. [31] ZHANG K H, LUO W H, ZHONG Y R, et al. Deblurring by Realistic Blurring // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2734-2743. [32] KIM K, LEE S, CHO S. MSSNet: Multi-scale-Stage Network for Single Image Deblurring // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 524-539. [33] ZHAO H Y, GOU Y B, LI B Y, et al. Comprehensive and Delicate: An Efficient Transformer for Image Restoration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 14122-14132. [34] CUI Y N, TAO Y, REN W Q, et al. Dual-Domain Attention for Image Deblurring. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(1): 479-487. [35] 杨浩,周冬明,赵倩. 结合梯度指导和局部增强Transformer的图像去模糊网络. 小型微型计算机系统, 2024, 45(1): 216-223. (YANG H, ZHOU D M, ZHAO Q. Image Deblurring Model Combining Gradient Guidance and Local Enhancement Transformer. Journal of Chinese Computer Systems, 2024, 45(1): 216-223.) [36] LIU C Z, HUA Z, LI J J. Reference-Based Dual-Task Framework for Motion Deblurring. The Visual Computer, 2023, 40(1): 137-151.
2024, Vol. 37 
