Image Super-Resolution Reconstruction Method Based on Heterogeneous Attention Network
ZHU Yujie1, ZHAO Jianwei1, LIU Jieyu1, ZHOU Zhenghua2
1. College of Information Engineering, China Jiliang University, Hangzhou 310018; 2. School of Data Sciences, Zhejiang University of Finance & Economics, Hangzhou 310018
Abstract：Existing Transformer-based image super-resolution reconstruction methods suffer from excessive computational redundancy due to their traditional multi-head design and dense self-attention mechanisms. To address this issue, an image super-resolution reconstruction method based on heterogeneous attention network is proposed in this paper. A heterogeneous multi-head self-attention block and a partial deep convolutional feedforward network block are designed. A three-branch structure is adopted in the heterogeneous multi-head self-attention block to reduce redundant computation. One dense branch for transmitting complete information is retained, a sparse branch to filter out noise is introduced, and a channel fusion branch is incorporated to supplement high-frequency information. Meanwhile, in the partial deep convolutional feedforward network block, the similarity in the activation feature maps is utilized to perform a partial deep convolutional operation to reduce the computational cost of the existing convolutional feedforward network block. Experimental results illustrate that the proposed method achieves better reconstruction performance with less computational cost.
[1] SHI W Z, CABALLERO J, HUSZAR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [2] LI Y W, AGUSTSSON E, GU S H, et al. CARN: Convolutional Anchored Regression Network for Fast and Accurate Single Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 166-181. [3] GENDY G, SABOR N, HE G H.Lightweight Image Super-Resolution Network Based on Extended Convolution Mixer. Engineering Applications of Artificial Intelligence, 2024, 133(A). DOI: 10.1016/j.engappai.2024.108069. [4] GAO G W, WANG Z X, LI J C, et al. Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer[C/OL].[2025-01-17]. https://arxiv.org/pdf/2204.13286. [5] DONG C, LOY C C, HE K M, et al. Learning a Deep Convolutio-nal Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199. [6] KIM J, LEE J K, LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654. [7] TONG T, LI G, LIU X J, et al. Image Super-Resolution Using Dense Skip Connections // Proc of the IEEE International Confe-rence on Computer Vision. Washington, USA: IEEE, 2017: 4809-4817. [8] KIM J, LEE J K, LEE K M.Deeply-Recursive Convolutional Network for Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1637-1645. [9] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2017: 1132-1140. [10] HUI Z, WANG X M, GAO X B.Fast and Accurate Single Image Super-Resolution via Information Distillation Network // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 723-731. [11] HUI Z, GAO X B, YANG Y C, et al. Lightweight Image Super-Resolution with Information Multi Distillation Network // Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 2024-2032. [12] 赵小强,李希尧,宋昭漾.轻量化逆可分离残差信息蒸馏网络的图像超分辨率重建.模式识别与人工智能, 2023, 36(5): 419-432. (ZHAO X Q, LI X Y, SONG Z Y.Lightweight Inverse Separable Residual Information Distillation Network for Image Super-Resolution Reconstruction. Pattern Recognition and Artificial Intelligence, 2023, 36(5): 419-432.) [13] BEHJATI P, RODRIGUEZ P, FERNÁNDEZ C, et al. Single Image Super-Resolution Based on Directional Variance Attention Network. Pattern Recognition, 2023, 133. DOI: 10.1016/j.patcog.2022.108997. [14] LUO X T, XIE Y, ZHANG Y L, et al. LatticeNet: Towards Lightweight Image Super-Resolution with Lattice Block // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 272-289. [15] FANG J S, LIN H J, CHEN X Y, et al. A Hybrid Network of CNN and Transformer for Lightweight Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2022: 1102-1111. [16] ZOU W B, YE T, ZHENG W X, et al. Self-Calibrated Efficient Transformer for Lightweight Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2022: 929-938. [17] CHEN H T, WANG Y H, GUO T Y, et al. Pre-trained Image Processing Transformer // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 12294-12305. [18] LU Z S, LI J C, LIU H, et al. Transformer for Single Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2022: 456-465. [19] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: Image Restoration Using Swin Transformer // Proc of the IEEE/CVF International Conference on Computer Vision Workshops. Washington, USA: IEEE, 2021: 1833-1844. [20] ZHANG X D, ZENG H, GUO S, et al. Efficient Long-Range Attention Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 649-667. [21] 王庭伟,赵建伟,周正华.基于轻量级对称CNN-Transformer的图像超分辨率重建方法.模式识别与人工智能, 2024, 37(7): 626-637. (WANG T W, ZHAO J W, ZHOU Z H.Image Super-Resolution Reconstruction Method Based on Lightweight Symmetric CNN-Transformer. Pattern Recognition and Artificial Intelligence, 2024, 37(7): 626-637.) [22] ZHOU Y P, LI Z, GUO C L, et al. SRFormer: Permuted Self-Attention for Single Image Super-Resolution // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2023: 12734-12745. [23] YUN S, RO Y.SHViT: Single-Head Vision Transformer with Me-mory Efficient Macro Design // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2024: 5756-5767. [24] ZHOU S H, CHEN D S, PAN J S, et al. Adapt or Perish: Adaptive Sparse Transformer with Attentive Feature Refinement for Ima-ge Restoration // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2024: 2952-2963. [25] WORTSMAN M, LEE J, GILMER J, et al. Replacing Softmax with ReLU in Vision Transformers[C/OL].[2025-01-17]. https://arxiv.org/pdf/2309.08586. [26] CHEN J R, KAO S H, HE H, et al. Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2023: 12021-12031. [27] WANG P H, ZHENG W Q, CHEN T L, et al. Anti-Oversmoo-thing in Deep Vision Trans Formers via the Fourier Domain Analysis: From Theory to Practice[C/OL].[2025-01-17]. https://arxiv.org/pdf/2203.05962. [28] PARK N, KIM S.How Do Vision Transformers Work?[C/OL]. [2025-01-17].https://arxiv.org/pdf/2202.06709. [29] AGUSTSSON E, TIMOFTE R.NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study // Proc of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington, USA: IEEE, 2017: 1122-1131. [30] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Complexity Single-Image Super-Resolution Based on Non-negative Neigh-bor Embedding[C/OL].[2025-01-17]. https://bmva-archive.org.uk/bmvc/2012/BMVC/paper135/paper135.pdf. [31] ZEYDE R, ELAD M, PROTTER M.On Single Image Scale-Up Using Sparse-Representations // Proc of the 7th International Conference on Curves and Surfaces. Berlin, Germany: Springer, 2012: 711-730. [32] MARTIN D, FOWLKES C, TAL D, et al. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics // Proc of the 8th IEEE International Conference on Computer Vision. Wa-shington, USA: IEEE, 2001: 416-423, 2001. [33] HUANG J B, SINGH A, AHUJA N.Single Image Super-Resolution from Transformed Self-Exemplars // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 5197-5206. [34] MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-Based Manga Retrieval Using Manga109 Dataset. Multimedia Tools and Applications, 2017, 76(20): 21811-21838.
2025, Vol. 38 
