Image Super-Resolution Reconstruction Method Based on Lightweight Symmetric CNN-Transformer
WANG Tingwei1, ZHAO Jianwei1, ZHOU Zhenghua2
1. College of Information Engineering, China Jiliang University, Hangzhou 310018; 2. School of Data Sciences, Zhejiang University of Finance & Economics, Hangzhou 310018
Abstract To address the issues of large parameter sizes and high training cost in existing image super-resolution reconstruction methods based on Transformer, an image super-resolution reconstruction method based on lightweight symmetric CNN-Transformer is proposed. Firstly, a symmetric CNN-Transformer block is designed using weight sharing, and the information extracted from the upper and lower branches is fully integrated through channel attention block to improve the ability of the network to capture and utilize both local and global features. Meanwhile, based on the depthwise separable convolution and the calculation of the self-attention cross-channel covariance matrix, the number of parameters in Transformer is effectively decreased, as well as calculation cost and memory consumption. Secondly, a high-frequency enhancement residual block is introduced into the network to further focus on the texture and detail information in the high-frequency area. Finally, the selection of the best activation function for generating the self-attention in Transformer is explored. Experimental analysis demonstrates that GELU function can better promote feature aggregation and improve network performance. Experimental results show that the proposed method effectively reconstructs richer textures and details of the image while maintaining the lightweight of the network.
Fund:Supported by Natural Science Foundation of Zhejiang Province(No.LY22F020002)
Corresponding Authors:
ZHAO Jianwei, Ph.D, professor. Her research interests include deep learning and image processing.
About author:: WANG Tingwei, Master student. His resear-ch interests include deep learning and image processing. ZHOU Zhenghua, Ph.D, associate profe-ssor. His research interests include deep lear-ning and image processing.
[1] GARCÍA-AGUILAR I, GARCÍA-GONZÁLEZ J, LUQUE-BAENA R M, et al. Automated Labeling of Training Data for Improved Object Detection in Traffic Videos by Fine-Tuned Deep Convolutional Neu-ral Networks. Pattern Recognition Letters, 2023, 167: 45-52. [2] DHAREJO F A, ZAWISH M, DEEBA F, et al. Multimodal-Boost: Multimodal Medical Image Super-Resolution Using Multi-attention Network with Wavelet Transform. IEEE/ACM Transactions on Com-putational Biology and Bioinformatics, 2023, 20(4): 2420-2433. [3] LIU Y F, XIONG Z T, YUAN Y, et al. Distilling Knowledge from Super-Resolution for Efficient Remote Sensing Salient Object Detection. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61. DOI: 10.1109/TGRS.2023.3267271. [4] DONG C, LOY C C, HE K M, et al. Learning a Deep Convolu-tional Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199. [5] DONG C, LOY C C, TANG X O. Accelerating the Super-Resolu-tion Convolutional Neural Network // Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2016: 391-407. [6] SHI W Z, CABALLERO J, HUSEÁR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [7] KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654. [8] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washing-ton, USA: IEEE, 2017: 5835-5843. [9] ZHANG Y L, LI K P, LI K, et al. Image Super-Resolution Using Very Deep Residual Channel Attention Networks // Proc of the Eu-ropean Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2018: 294-310. [10] ZHANG Y L, TIAN Y P, KONG Y, et al. Residual Dense Net-work for Image Super-Resolution // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2472-2481. [11] MEI Y Q, FAN Y C, ZHOU Y Q. Image Super-Resolution with Non-local Sparse Attention // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 3516-3525. [12] WU X, ZHANG K B, HU Y T, et al. Multi-scale Non-local Atten-tion Network for Image Super-Resolution. Signal Processing, 2024, 218. DOI: 10.1016/j.sigpro.2023.109362. [13] KIM J, LEE J K, LEE K M. Deeply-Recursive Convolutional Network for Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1637-1645. [14] TAI Y, YANG J, LIU X M. Image Super-Resolution via Deep Re-cursive Residual Network // Proc of the IEEE Conference on Com-puter Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2790-2798. [15] 张大宝,赵建伟,周正华.基于l1诱导轻量级深度网络的图像超分辨率重建.模式识别与人工智能, 2022, 35(12): 1101-1110. (ZHANG D B, ZHAO J W, ZHOU Z H. Image Super-Resolution Reconstruction Based on l1 Induced Lightweight Deep Networks. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1101-1110.) [16] HUI Z, GAO X B, YANG Y C, et al. Lightweight Image Super-Resolution with Information Multi-distillation Network // Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 2024-2032. [17] ZHA L, YANG Y, LAI Z C, et al. A Lightweight Dense Connec-ted Approach with Attention on Single Image Super-Resolution. Electronics, 2021, 10(11). DOI: 10.3390/electronics10111234. [18] LAN R S, SUN L, LIU Z B, et al. MADNet: A Fast and Light-weight Network for Single-Image Super Resolution. IEEE Transa-ctions on Cybernetics, 2021, 51(3): 1443-1453. [19] PENG C M, SHU P, HUANG X Y, et al. LCRCA: Image Super-Resolution Using Lightweight Concatenated Residual Channel Attention Networks. Applied Intelligence, 2022, 52(9): 10045-10059. [20] 周登文,王婉君,马钰,等. 基于区域互补注意力和多维注意力的轻量级图像超分辨率网络.模式识别与人工智能, 2022, 35(7): 625-636. (ZHOU D W, WANG W J, MA Y, et al. Lightweight Image Super-Resolution Network Based on Regional Complementary Attention and Multi-dimensional Attention. Pattern Recognition and Artificial Intelligence, 2022, 35(7): 625-636.) [21] GAO D D, ZHOU D W. A Very Lightweight and Efficient Image Super-Resolution Network. Expert Systems with Applications, 2023, 213. DOI: 10.1016/j.eswa.2022.118898. [22] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010. [23] LI Y W, ZHANG K, CAO J Z, et al. LocalViT: Bringing Locality to Vision Transformers[C/OL].[2024-03-16]. https://arxiv.org/pdf/2104.05707. [24] LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 9992-10002. [25] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: Image Restoration Using Swin Transformer // Proc of the IEEE/CVF Interna-tional Conference on Computer Vision. Washington, USA: IEEE, 2021: 1833-1844. [26] WANG W, ZHU Y F, DING D W, et al. Multi-scale Multi-stage Single Image Super-Resolution Reconstruction Algorithm Based on Transformer // Proc of the 21st International Symposium on Distri-buted Computing and Applications for Business Engineering and Science. Washington, USA: IEEE, 2022: 111-114. [27] LU Z S, LI J C, LIU H, et al. Transformer for Single Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 456-465. [28] FANG J S, LIN H J, CHEN X Y, et al. A Hybrid Network of CNN and Transformer for Lightweight Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 1102-1111. [29] LI X, DONG J X, TANG J H, et al. DLGSANet: Lightweight Dynamic Local and Global Self-Attention Network for Image Super-Resolution // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2023: 12746-12755. [30] ZAMIR S W, ARORA A, KHAN S, et al. Restormer: Efficient Transformer for High-Resolution Image Restoration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 5718-5729. [31] HU J, SHEN L, SUN G. Squeeze-and-Excitation Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2018: 7132-7141. [32] LI A, ZHANG L, LIU Y, et al. Feature Modulation Transformer:Cross-Refinement of Global Representation via High-Frequency Prior for Image Super-Resolution // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2023: 12480-12490. [33] LIU J, ZHANG W J, TANG Y T, et al. Residual Feature Aggregation Network for Lightweight Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2356-2365. [34] TIMOFTE R, AGUSTSSON E, VAN GOOL L, et al. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results // Proc of the IEEE Conference on Computer and Pattern Recognition Workshops. Washington, USA: IEEE, 2017: 1110-1121. [35] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Com-plexity Single-Image Super-Resolution Based on Nonnegative Neigh-bor Embedding // Proc of the British Machine Vision Conference. Bristol, UK: BMVA, 2012. DOI: 10.5244/C.26.135. [36] ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-Up Using Sparse-Representations // Proc of the 7th International Con-ference on Curves and Surfaces. Berlin, Germany: Springer, 2010: 711-730. [37] ARBELÁEZ P, MAIRE M, FOWLKES C, et al. Contour Detec-tion and Hierarchical Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 898-916. [38] HUANG J B, SINGH A, AHUJA N. Single Image Super-Resolu-tion from Transformed Self-Exemplars // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 5197-5206. [39] MATSUI Y, ITO K, ARAMAKI Y,et al. Sketch-Based Manga Retrieval Using Manga109 Dataset. Multimedia Tools and Appli-cations, 2017, 76(20): 21811-21838. [40] GAO X B, LU W, TAO D C, et al. Image Quality Assessment Based on Multiscale Geometric Analysis. IEEE Transactions on Image Processing, 2009, 18(7): 1409-1423. [41] KINGMA D P, BA J L. Adam: A Method for Stochastic Optimi-zation[C/OL]. [2024-03-16].https://arxiv.org/pdf/1412.6980. [42] HUI Z, WANG X M, GAO X B. Fast and Accurate Single Image Super-Resolution via Information Distillation Network // Proc of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 723-731. [43] ZHAO H Y, KONG X T, HE J W, et al. Efficient Image Super-Resolution Using Pixel Attention // Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2020: 56-72. [44] SUN Z F, ZHAO J W, ZHOU Z H, et al. L1 Model-Driven Recur-sive Multi-scale Denoising Network for Image Super-Resolution. Know-ledge-Based Systems, 2021, 225(1). DOI: 10.1016/j.knosys.2021.107115. [45] WANG L G, DONG X Y, WANG Y Q, et al. Exploring Sparsity in Image Super-Resolution for Efficient Inference // Proc of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 4915-4924. [46] PANG S R, CHEN Z, YIN F L. Image Super-Resolution Based on Generalized Residual Network. Arabian Journal for Science and Engineering, 2022, 47(2): 1903-1920. [47] 赵小强,李希尧,宋昭漾.轻量化逆可分离残差信息蒸馏网络的图像超分辨率重建.模式识别与人工智能, 2023, 36(5): 419-432. (ZHAO X Q, LI X Y, SONG Z Y. Lightweight Inverse Separable Residual Information Distillation Network for Image Super-Resolu-tion Reconstruction. Pattern Recognition and Artificial Intelligence, 2023, 36(5): 419-432.)
2024, Vol. 37 
