Abstract While low-resolution face images are reconstructed via deep learning based super-resolution reconstruction method, some problems emerge, such as blurred reconstructed images and obvious difference between reconstructed images and real images. Aiming at these problems, a face super-resolution reconstruction method fusing reference image is proposed to reconstruct low-resolution human face images effectively. The multi-scale features of reference image are extracted by reference image feature extraction subnet to retain the detail information of key parts and remove the redundant information, such as facial contour and facial expression. Based on the multi-scale features of reference image, the step-by-step super-resolution main network fills the features to low-resolution face image step by step. Finally, the high-resolution face image is generated. Experiments on datasets indicate that the proposed method reconstructs low-resolution face images effectively with good robustness.
Fund:Supported by Natural Science Foundation of Beijing(No.4173072)
About author:: FU Lihua, Ph.D., associate professor. Her research interests include intelligent information processing, image processing and computer vision.LU Zhongshan, master student. His research interests include computer image processing and super resolution.SUN Xiaowei, master student. His research interests include computer image processing and super resolution.ZHAO Yu, master student. His research interests include computer image processing and video target segmentation.ZHANG Bo, master student. His research interests include computer image processing and video target segmentation.
[1] 张 地,何家忠.基于特征空间的人脸超分辨率重构.自动化学报, 2012, 38(7): 1145-1152. (ZHANG D, HE J Z. Feature Space Based Face Super-Resolution Reconstruction. Acta Automatica Sinica, 2012, 38(7): 1145-1152.) [2] 唐佳林,吴泽锋,蒋才高,等.基于小波变换的图像超分辨率复原算法研究.计算机科学, 2014, 41(11A): 147-149. (TANG J L, WU Z F, JIANG C G, et al. Study of Super-Resolution Image Restoration Algorithm Based on Wavelet Transform. Computer Science, 2014, 41(11A): 147-149.) [3] LI X, ORCHARD M T. New Edge-Directed Interpolation. IEEE Transactions on Image Processing, 2001, 10(10): 1521-1527. [4] HWANG J W, LEE H S. Adaptive Image Interpolation Based on Local Gradient Features. IEEE Signal Processing Letters, 2004, 11(3): 359-362. [5] 王春霞,苏红旗,范郭亮.图像超分辨率重建技术综述.计算机技术与发展, 2011, 21(5): 124-127. (WANG C X, SU H Q, FANG G L. Overview on Super Resolution Image Reconstruction. Computer Technology and Development, 2011, 21(5): 124-127.) [6] DONG C, LOY C C, HE K M, et al. Learning a Deep Convolutio-nal Network for Image Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199. [7] DONG C, LOY C C, TANG X O. Accelerating the Super-Resolution Convolutional Neural Network // Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2016: 391-407. [8] HE K M, ZHANG X Y, REN S Q, et al. Identity Mappings in Deep Residual Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 630-645. [9] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 136-144. [10] YU J H, FAN Y C, YANG J C, et al. Wide Activation for Efficient and Accurate Image Super-resolution[C/OL]. [2019-10-26]. https://arxiv.org/pdf/1808.08718.pdf. [11] ZHANG Y L, TIAN Y P, KONG Y, et al. Residual Dense Network for Image Super-Resolution // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2472-2481. [12] YOO J C, AHN C W. Image Matching Using Peak Signal-to-Noise Ratio-Based Occlusion Detection. IET Image Processing, 2012, 6(5): 483-495. [13] SAMPAT M P, WANG Z, GUPTA S, et al. Complex Wavelet Structural Similarity: A New Image Similarity Index. IEEE Transactions on Image Processing, 2009, 18(11): 2385-2401. [14] YU X, PORIKLI F. Ultra-Resolving Face Images by Discriminative Generative Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 318-333. [15] YU X, PORIKLI F. Hallucinating Very Low-Resolution Unaligned and Noisy Face Images by Transformative Discriminative Autoencoders // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 3760-3768. [16] CHEN Y, TAI Y, LIU X, et al. FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2492-2501. [17] BEN X Y, GONG C, ZHANG P, et al. Coupled Patch Alignment for Matching Cross-View Gaits. IEEE Transactions on Image Processing, 2019, 28(6): 3142-3157. [18] VU T, WAN NGUYEN C, PHAM T X, et al. Fast and Efficient Image Quality Enhancement via Desubpixel Convolutional Neural Networks // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 243-259. [19] 李现国,孙叶美,杨彦利,等.基于中间层监督卷积神经网络的图像超分辨率重建.中国图象图形学报, 2018, 23(7): 984-993. (LI X G, SUN Y M, YANG Y L, et al. Image Super-Resolution Reconstruction Based on Intermediate Supervision Convolutional Neural Networks. Journal of Image and Graphics, 2018, 23(7): 984-993,) [20] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-Excitation Networks // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 7132-7141. [21] ZHOU K Y, YANG Y X, CAVALLARO A, et al. Omni-Scale Feature Learning for Person Re-identification[C/OL]. [2019-10-26]. https://arxiv.org/pdf/1905.00953.pdf. [22] ZHANG Z F, WANG Z W, LIN Z, et al. Image Super-Resolution by Neural Texture Transfer // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 7982-7991. [23] SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-pixel Convolutional Neural Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1874-1883. [24] GATYS L A, ECKER A S, BETHGE M. A Neural Algorithm of Artistic Style[C/OL]. [2019-10-26]. https://arxiv.org/pdf/1508.06576.pdf. [25] JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711. [26] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Gene-rative Adversarial Nets // GHAHRAMANI Z, WELLING M, CORTES C, et al., eds. Advances in Neural Information Proce-ssing Systems 27. Cambridge, USA: The MIT Press, 2014: 2672-2680. [27] LE V, BRANDT J, LIN Z, et al. Interactive Facial Feature Loca-lization // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2012: 679-692. [28] RIM D, HASAN M K, PUECH F, et al. Learning from Weakly Labeled Faces and Video in the Wild. Pattern Recognition, 2015, 48(3): 759-771. [29] LIU Z W, LUO P, WANG X O, et al. Deep Learning Face Attri-butes in the Wild // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 3730-3738. [30] NEWEY W K. Adaptive Estimation of Regression Models via Moment Restrictions. Journal of Econometrics, 1988, 38(3): 301-339. [31] BOTTOU L. Large-Scale Machine Learning with Stochastic Gradient Descent // Proc of COMPSTAT'2010. Berlin, Germany: Springer, 2010: 177-186.
2020, Vol. 33 
