Abstract Lightweight convolution neural networks embody the advantages in small parameters, low computational cost and fast reasoning speed. However, the performance of the networks is greatly limited. To improve the performance of the lightweight image super-resolution network, a lightweight image super-resolution network based on regional complementary attention and multi-dimensional attention is proposed. Its basic component,dual branch multiple interactive residual block, can fuse multi-scale features effectively. To improve the utilization and expression ability of features, effective lightweight region complementary attention is designed to make the information in different regions of the feature map complement each other. Multi-dimensional attention is designed to model the dependencies between pixels in channel and spatial dimensions. Experimental results demonstrate that the proposed network is superior to the current lightweight super-resolution methods in complexity and performance balance.
Corresponding Authors:ZHOU Dengwen, master, professor. His research interests include image denoising, image demosaic, image interpolation and image super-resolution.
About author:: About Author:WANG Wanjun, master student. Her research interests include computer vision and deep learning.
MA Yu, master student. His research interests include computer vision and deep learning.
GAO Dandan, master student. Her research interests include computer vision and deep learning.
[1] FREEMAN W T, PASZTOR E C. Learning Low-Level Vision // Proc of the 7th IEEE International Conference on Computer Vision. Washington, USA: IEEE, 1999, II: 1182-1189. [2] SHI W Z, CABALLERO J, LEDIG C, et al. Cardiac Image Super-Resolution with Global Correspondence Using Multi-atlas PatchMatch // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2013: 9-16. [3] SHERMEYER J, VAN ETTEN A. The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2019: 1432-1441. [4] ZOU W W W, YUEN P C. Very Low Resolution Face Recognition Problem. IEEE Transactions on Image Processing, 2012, 21(1): 327-340. [5] THORNTON M W, ATKINSON P M, HOLLAND D A. Sub-Pixel Mapping of Rural Land Cover Objects from Fine Spatial Resolution Satellite Sensor Imagery Using Super-Resolution Pixel-Swapping. International Journal of Remote Sensing, 2006, 27(3): 473-491. [6] ZHANG L, WU X L. An Edge-Guided Image Interpolation Algorithm via Directional Filtering and Data Fusion. IEEE Transactions on Image Processing, 2006, 15(8): 2226-2238. [7] JENSEN K, ANASTASSIOU D. Subpixel Edge Localization and the Interpolation of Still Images. IEEE Transactions on Image Proce-ssing, 1995, 4(3): 285-295. [8] YANG J C, WRIGHT J, HUANG T S, et al. Image Super-Resolution via Sparse Representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. [9] ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-Up Using Sparse-Representations // Proc of the International Confe-rence on Curves and Surfaces. Berlin, Germany: Springer, 2010: 711-730. [10] YANG J C, WANG Z W, LIN Z, et al. Coupled Dictionary Training for Image Super-Resolution. IEEE Transactions on Image Processing, 2012, 21(8): 3467-3478. [11] DONG C, LOY C C, HE K M, et al. Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. [12] LI Z, YANG J L, LIU Z, et al. Feedback Network for Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 3862-3871. [13] YANG X, MEI H Y, ZHANG J Q, et al. DRFN: Deep Recurrent Fusion Network for Single-Image Super-Resolution with Large Factors. IEEE Transactions on Multimedia, 2019, 21(2): 328-337. [14] HE Z W, CAO Y P, DU L, et al. MRFN: Multi-receptive-Field Network for Fast and Accurate Single Image Super-Resolution. IEEE Transactions on Multimedia, 2020, 22(4): 1042-1054. [15] HU Y T, LI J, HUANG Y F, et al. Channel-Wise and Spatial Feature Modulation Network for Single Image Super-Resolution. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(11): 3911-3927. [16] DAI T, CAI J R, ZHANG Y B, et al. Second-Order Attention Network for Single Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 11057-11066. [17] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2017: 1132-1440. [18] ZHANG Y L, LI K P, LI K, et al. Image Super-Resolution Using Very Deep Residual Channel Attention Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2018: 294-310. [19] KIM J, LEE J K, LEE K M. Deeply-Recursive Convolutional Network for Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1637-1645. [20] TAI Y, YANG J, LIU X M. Image Super-Resolution via Deep Recursive Residual Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2790-2798. [21] TAI Y, YANG J, LIU X M, et al. MemNet: A Persistent Memory Network for Image Restoration // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4549-4557. [22] LI W B, LI J F, LI J X, et al. A Lightweight Multi-scale Channel Attention Network for Image Super-Resolution. Neurocomputing, 2021, 456: 327-337. [23] 周登文,赵丽娟,段然,等.基于递归残差网络的图像超分辨率重建.自动化学报, 2019, 45(6): 1157-1165. (ZHOU D W, ZHAO L J, DUAN R, et al. Image Super-Resolution Based on Recursive Residual Networks. Acta Automatica Sinica, 2019, 45(6): 1157-1165.) [24] 陈一鸣,周登文. 基于自适应级联的注意力网络的超分辨重建[J/OL]. [2020-12-30].https://doi.org/10.16383/j.aas.c200035. (CHEN Y M, ZHOU D W. Adaptive Attention Network for Image Super-Resolution[J/OL]. [2020-12-30].https://doi.org/10.16383/j.aas.c200035.) [25] HUI Z, GAO X B, YANG Y C, et al. Lightweight Image Super-Resolution with Information Multi-distillation Network // Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 2024-2032. [26] LIU J, TANG J, WU G S. Residual Feature Distillation Network for Lightweight Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 41-55. [27] ZHANG K, DANELLJAN M, LI Y W, et al. AIM 2020 Challenge on Efficient Super-Resolution: Methods and Results // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 5-40. [28] LI W B, ZHOU K, QI L, et al. LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond[C/OL].[2022-03-25]. https://arxiv.org/pdf/2105.10422.pdf. [29] ZHAO H Y, KONG X T, HE J W, et al. Efficient Image Super-Resolution Using Pixel Attention // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 56-72. [30] CHEN H Y, GU J J, ZHANG Z. Attention in Attention Network for Image Super-Resolution[C/OL]. [2022-03-25]. https://arxiv.org/pdf/2104.09497v3.pdf. [31] 李金新,黄志勇,李文斌,等. 基于多层次特征融合的图像超分辨率重建[J/OL]. [2020-12-30]. https://doi.org/10.16383/j.aas.c200585. (LI J X, HUANG Z Y, LI W B, et al. Image Super-Resolution Based on Multi Hierarchical Features Fusion Network[J/OL].[2020-12-30]. https://doi.org/10.16383/j.aas.c200585.) [32] LI J C, FANG F M, MEI K F, et al. Multi-scale Residual Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 527-542. [33] MAAS A L, HANNUN A Y, NG A Y. Rectifier Nonlinearities Improve Neural Network Acoustic Models[C/OL]. [2022-03-25]. http://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf. [34] SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [35] ZHANG Y L, TIAN Y P, KONG Y, et al. Residual Dense Network for Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2472-2481. [36] ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 6848-6856. [37] IOFFE S, SZEGEDY C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift // Proc of the 32nd International Conference on Machine Learning. San Diego, USA: JMLR, 2015: 448-456. [38] AGARAP A F M. Deep Learning Using Rectified Linear Units[C/OL].[2022-03-25]. https://arxiv.org/pdf/1803.08375.pdf. [39] ZHANG J Q, LONG C J, WANG Y X, et al. A Two-Stage Attentive Network for Single Image Super-Resolution. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(3): 1020-1033. [40] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023. [41] WOO S, PARK J, LEE J Y, et al. CBAM: Convolutional Block Attention Module // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19. [42] TIMOFTE R, AGUSTSSON E, VAN GOOL L, et al. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results // Proc of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2017: 1110-1121. [43] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Com-plexity Single-Image Super-Resolution Based on Nonnegative Neighbor Embedding // Proc of the 23rd British Machine Vision Confe-rence. Guildford, UK: BMVA Press, 2012. DOI: 10.5244/C.26.135. [44] MARTIN D, FOWLKES C, TAL D, ,et al. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics // Proc of the 8th IEEE International Conference on Computer Vision. Wa-shington, USA: IEEE, 2001, II: 416-423. [45] HUANG J B, SINGH A, AHUJA N. Single Image Super-Resolution from Transformed Self-Exemplars // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 5197-5206. [46] MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-Based Manga Retrieval Using Manga109 Dataset. Multimedia Tools and Applications, 2017, 76(20): 21811-21838. [47] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. [48] KINGMA D P, BA J L. Adam: A Method for Stochastic Optimization[C/OL]. [2022-03-25]. https://arxiv.org/pdf/1412.6980.pdf. [49] PASZKE A, GROSS S, CHINTALA S, et al. Automatic Differentiation in PyTorch[C/OL].[2022-03-25]. https://openreview.net/pdf?id=BJJsrmfCZ. [50] AHN N, KANG B, SOHN K A. Fast, Accurate, Lightweight Super-Resolution with Cascading Residual Network // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 256-272. [51] HUI Z, WANG X M, GAO X B. Fast and Accurate Single Image Super-Resolution via Information Distillation Network // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recogni-tion. Washington, USA: IEEE, 2018: 723-731. [52] DONG C, LOY C C, TANG X. Accelerating the Super-Resolution Convolutional Neural Network // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 391-407. [53] KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654. [54] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5835-5843. [55] WANG C F, LI Z, SHI J. Lightweight Image Super-Resolution with Adaptive Weighted Learning Network[C/OL]. [2022-03-25]. https://arxiv.org/ftp/arxiv/papers/1904/1904.02358.pdf. [56] WANG L G, DONG X Y, WANG Y Q, et al. Exploring Sparsity in Image Super-Resolution for Efficient Inference // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 4915-4924.
2022, Vol. 35 
