Structure-Preserving Super-Resolution Reconstruction Based on Multi-residual Network
ZHANG Mingjin1,2, PENG Xiaoqi1, GUO Jie1, LI Yunsong1, WANG Nannan1, GAO Xinbo1,3
1. State Key Laboratory of Integrated Services Networks, Xidian University, Xi'an 710071 2. Key Laboratory of Spectral Imaging Technology, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an 710119 3. Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing 400065
Abstract Aiming at the problems of geometric structure distortion and missing details in image super-resolution reconstruction, a structure-preserving super-resolution reconstruction algorithm based on multi-residual network is proposed. Deep learning is carried out in the wavelet transform domain and the gradient domain. Three kinds of residual networks are introduced. The structure and the edge information are reconstructed by the residual gradient network. The high-frequency information of the image is reconstructed as a whole by the residual wavelet transform network. The network attention is adjusted by the residual channel attention network , the important channel features are emphatically learned, and the high frequency information of the image is recovered locally. Experiments show that the proposed algorithm achieves better performance in both quantitative results and visual effects.
Fund:Youth Fund of National Natural Science Foundation of China(No.61902293), Young Talent Fund of University Association for Science and Technology in Shaanxi(No.20200103), Open Research Fund of CAS Key Laboratory of Spectral Imaging Technology(No.LSIT201901W), Fundamental Research Funds for the Central Universities(No.XJS200112)
Corresponding Authors:
GAO Xinbo, Ph.D., professor. His research interests include pa-ttern recognition and computer vision.
About author:: ZHANG Mingjin, Ph.D., associate professor. Her research interests include pattern recognition and computer vision.PENG Xiaoqi, master student. Her research interests include pattern recognition and computer vision.GUO Jie, Ph.D., associate professor. His research interests include image transmission and processing.LI Yunsong, Ph.D., professor. His research interests include image transmission and processing.WANG Nannan, Ph.D., professor. His research interests include pattern recognition and computer vision.
[1] 付利华,孙晓威,赵 宇,等.基于运动特征融合的快速视频超分辨率重构方法.模式识别与人工智能, 2019, 32(11): 1022-1031. (FU L H, SUN X W, ZHAO Y, et al. Fast Video Super-Resolution Reconstruction Method Based on Motion Feature Fusion. Pattern Recognition and Artificial Intelligence, 2019, 32(11): 1022-1031. [2] LEDIG C, THEIS L, HUSZÁR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 105-114. [3] DONG C, CHEN C L, TANG X O. Accelerating the Super-Resolution Convolutional Neural Network // Proc of the 14th European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 391-407. [4] 付利华,卢中山,孙晓威,等.融合参考图像的人脸超分辨率重构方法.模式识别与人工智能, 2020, 33(4): 325-336. (FU L H, LU Z S, SUN X W, et al. Face Super-Resolution Reconstruction Method Fusing Reference Image. Pattern Recognition and Artificial Intelligence, 2020, 33(4): 325-336.) [5] SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [6] ZOU W W W, YUEN P C. Very Low Resolution Face Recognition Problem // Proc of the 4th IEEE International Conference on Biometrics: Theory, Applications and Systems. Washington, USA: IEEE, 2010. DOI: 10.1109/BTAS.2010.5634490. [7] SHI W Z, CABALLERO J, LEDIG C, et al. Cardiac Image Super-Resolution with Global Correspondence Using Multi-atlas PatchMatch // Proc of the International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2013: 9-16. [8] 郑少飞,汤 进,罗 斌,等.基于改进损失函数的多阶段行人属性识别方法.模式识别与人工智能, 2018, 31(12): 1085-1095. (ZHENG S F, TANG J, LUO B, et al. Multistage Pedestrian Attribute Recognition Method Based on Improved Loss Function. Pattern Recognition and Artificial Intelligence, 2018, 31(12): 1085-1095.) [9] 蒋桧慧,张 荣,李小宝,等.融合直接度量和间接度量的行人再识别.模式识别与人工智能, 2018, 31(2): 167-174. (JIANG H H, ZHANG R, LI X B, et al. Pedestrian Re-identification Fusing Direct Metric and Indirect Metric. Pattern Recognition and Artificial Intelligence, 2018, 31(2): 167-174.) [10] 刘 丽,郑 洋,付冬梅.改进YOLOv3网络结构的遮挡行人检测算法.模式识别与人工智能, 2020, 33(6): 568-574. (LIU L, ZHENG Y, FU D M, et al. Occluded Pedestrian Detection Algorithm Based on Improved Network Structure of YOLOv3. Pattern Recognition and Artificial Intelligence, 2020, 33(6): 568-574.) [11] DONG C, LOY C C, HE K M, et al. Learning a Deep Convolutional Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199. [12] KIM J, LEE J K, LEE K M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654. [13] TAI Z, YANG J, LIU X M. Image Super-Resolution via Deep Recursive Residual Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2790-2798. [14] KIM J, LEE J K, LEE K M. Deeply-Recursive Convolutional Network for Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1637-1645. [15] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [16] LIM B, SON S, KIM M, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 136-144. [17] TAI Y, YANG J, LIU X M, et al. MemNet: A Persistent Memory Network for Image Restoration // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4549-4557. [18] CHOI J S, KIM M. A Deep Convolutional Neural Network with Selection Units for Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1150-1156. [19] LI J C, FANG F M, MEI K F, et al. Multi-scale Residual Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 527-542. [20] AHN N, KONG B, SOHN K A. Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 252-268. [21] HE X Y, MO Z T, WANG P S, et al. Ode-Inspired Network Design for Single Image Super-Resolution // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1732-1741. [22] ZHANG Y L, LI K P, LI K, et al. Image Super Resolution Using Very Deep Residual Channel Attention Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Sprin-ger, 2018: 294-310. [23] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5835-5843. [24] SUN J, XU Z B, SHUM H Y. Gradient Profile Prior and Its Applications in Image Super-Resolution and Enhancement. IEEE Transactions on Image Processing, 2010, 20(6): 1529-1542. [25] KUMAR N, VERMA R, SETHI A. Convolutional Neural Networks for Wavelet Domain Super Resolution. Pattern Recognition Letters, 2017, 90: 65-71. [26] TIMOFTE R, AGUSTSSON E, VAN GOOL L, et al. NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1110-1121. [27] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-Complexity Single-Image Super-Resolution Based on Nonnegative Neigh-bor Embedding // Proc of the British Machine Vision Conference. Berlin, Germany: Springer, 2012: 1-10. [28] DE SOUZA FARIA G, KIM H Y. Differential Audio Analysis: A New Side-Channel Attack on Pin Pads. International Journal of Information Security, 2019, 18: 73-84. [29] MARTIN D, FOWLKES C C, TAL D, et al. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics // Proc of the International Conference on Computer Vision. Washington, USA: IEEE, 2001: 416-423. [30] HUANG J B, SINGH A, AHUJA N. Single Image Super-Resolution from Transformed Self-exemplars // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 5197-5206. [31] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
2021, Vol. 34 
