基于多任务对抗和抗噪对抗学习的人脸超分辨率算法

doi:10.16451/j.cnki.issn1003-6059.202210001

Abstract
Figure/Table
References (0)
Related Citation (15)

Download: PDF (2122 KB) HTML (1 KB)
Export: BibTeX | EndNote (RIS)

Abstract

The super-resolution(SR) of high magnification single face image is a hard but valuable task. In the face super-resolution(FSR) task, the end-to-end network SR image is fuzzy, and the photoreality and human visual effect are poor. Aiming at the problems, a FSR algorithm based on multi-task adversarial learning(MTAL) and antinoise adversarial learning(ANAL) is proposed. The algorithm is divided into end-to-end network learning and network parameters fine-tuning. To improve the end-to-end learning result, a deep multi-task Laplacian pyramid network(MTLapNet) is designed and integrated with MTAL. The main task is end-to-end learning, while the subtask is the optimization of adversarial learning penalty function. To improve the result of adversarial learning and parameters fine-tuning of the main task network, ANAL is integrated into the optimization process of discriminator of adversarial learning. The experiments show that the proposed algorithm can make the FSR image more photo-realistic and more consistent with human visual habits.

Key words： Deep Learning Face Super-Resolution(FSR) Multi-task Adversarial Learning(MTAL) Antinoise Adversarial Learning(ANAL) Multi-task Laplacian Pyramid Network(MTLapNet)

Received: 18 April 2022

ZTFLH:	TP 391.41
	TP 183

Fund:

National Natural Science Foundation of China(No.U1936113,61872303)

Corresponding Authors: HE Hongjie, Ph.D., professor. Her research interests include image forensics and image processing.

About author:: CHEN Hongyou, Ph.D. His research interests include machine learning and image processing.CHEN Fan, Ph.D., associate professor. His research interests include multi-media security and computer applications.JIANG Tongyu, master student. His research interests include face super-resolution.

	Service

	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	CHEN Hongyou
	CHEN Fan
	HE Hongjie
	JIANG Tongyu

Cite this article:

CHEN Hongyou,CHEN Fan,HE Hongjie等. Face Super-Resolution Algorithm Based on Multi-task Adversarial and Antinoise Adversarial Learning[J]. Pattern Recognition and Artificial Intelligence, 2022, 35(10): 863-880.

URL:

http://manu46.magtech.com.cn/Jweb_prai/EN/10.16451/j.cnki.issn1003-6059.202210001 OR http://manu46.magtech.com.cn/Jweb_prai/EN/Y2022/V35/I10/863

[1] CHEN H G, HE X H, QING L B, et al. Real-World Single Image Super-Resolution: A Brief Review. Information Fusion, 2022, 79: 124-145.
[2] ZHANG M J, XIN J W, ZHANG J, et al. Curvature Consistent Network for Microscope Chip Image Super-Resolution. IEEE Transactions on Neural Network and Learning Systems, 2022. DOI: 10.1109/TNNLS.2022.3168540.
[3] DONG C, LOY C C, HE K M, et al. Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[4] KIM J, LEE J K, LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654.
[5] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5835-5843.
[6] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1132-1140.
[7] LI Z, YANG J L, LIU Z, et al. Feedback Network for Image Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 3862-3871.
[8] ZHAO X L, LIAO Y, HE T, et al. FC²N: Fully Channel-Concatenated Network for Single Image Super-Resolution[C/OL].[2021-03-15]. https://arxiv.org/pdf/1907.03221.pdf.
[9] LIU J, ZHANG W J, TANG Y T, et al. Residual Feature Aggregation Network for Image Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2356-2365.
[10] 张铭津,彭晓琪,郭杰,等.基于多残差网络的结构保持超分辨重建.模式识别与人工智能, 2021, 34(3): 232-240.
(ZHANG M J, PENG X Q, GUO J, et al. Structure-Preserving Super-Resolution Reconstruction Based on Multi-residual Network. Pattern Recognition and Artificial Intelligence, 2021, 34(3): 232-240.)
[11] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: Image Restoration Using Swin Transformer// Proc of the IEEE/CVF Internatio-nal Conference on Computer Vision Workshops. Washington, USA: IEEE, 2021: 1833-1844.
[12] ZHANG M J, WU Q Q, ZHANG J, et al. Fluid Micelle Network for Image Super-Resolution Reconstruction. IEEE Transactions on Cybernetics, 2022. DOI: 10.1109/TCYB.2022.3163294.
[13] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al.Ge-nerative Adversarial Nets// Proc of the 27th International Confe-rence on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2014: 2672-2680.
[14] HONG Y J, HWANG U, YOO J, et al. How Generative Adversa-rial Networks and Their Variants Work: An Overview. ACM Computing Surveys, 2019, 52(1). DOI: 10.1145/3301282.
[15] LEDIG C, THEIS L, HUSZÁR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network// Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 105-114.
[16] WANG X T, YU K, WU S X, et al. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks// Proc of the Euro-pean Conference on Computer Vision. Berlin, Germany: Springer, 2018: 63-79.
[17] RAKOTONIRINA N C, RASOANAIVO A.ESRGAN+: Further Improving Enhanced Super-Resolution Generative Adversarial Network// Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington, USA: IEEE, 2020: 3637-3641.
[18] HARIS M, SHAKHNAROVICH G, UKITA N.Deep Back-Projection Networks for Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 1664-1673.
[19] HUI Z, LI J, GAO X B, et al. Progressive Perception-Oriented Network for Single Image Super-Resolution. Information Sciences, 2021, 546: 769-786.
[20] JIANG J J, WANG C Y, LIU X M, et al. Deep Learning-Based Face Super-Resolution: A Survey. ACM Computing Surveys, 2003, 55(1). DOI: 10.1145/3485132.
[21] LIU H, ZHENG X Y, HAN J G, et al. Survey on GAN-Based Face Hallucination with Its Model Development. IET Image Processing, 2019, 13(14): 2662-2672.
[22] GOSWAMI S, AAKANKSH A, RAJAGOPALAN A N. Robust Super-Resolution of Real Faces Using Smooth Features// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 169-185.
[23] LI X M, CHEN C F, ZHOU S C, et al. Blind Face Restoration via Deep Multi-scale Component Dictionaries// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 399-415.
[24] CHEN Y, TAI Y, LIU X M, et al. FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2492-2501.
[25] MA C, JIANG Z Y, RAO Y M, et al. Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation// Proc of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 5568-5577.
[26] ZHANG Y C, WU Y, CHEN L.MSFSR: A Multi-stage Face Super-Resolution with Accurate Facial Representation via Enhanced Facial Boundaries// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2120-2129.
[27] 蒋桐雨,陈帆,和红杰.基于非对称U型金字塔重建的轻量级人脸超分辨率网络.山东大学学报 (工学版), 2022, 52(1): 1-8, 18.
(JIANG T Y, CHEN F, HE H J.Lightweight Face Super-Resolution Network Based on Asymmetric U-Pyramid Reconstruction. Journal of Shandong University (Engineering Science), 2022, 52(1): 1-8, 18.)
[28] KIM D, KIM M, KWON G, et al. Progressive Face Super-Resolution via Attention to Facial Landmark[C/OL].[2022-03-15]. https://arxiv.org/pdf/1908.08239v1.pdf.
[29] BULAT A, TZIMIROPOULOS G.Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 109-117.
[30] 付利华,卢中山,孙晓威,等.融合参考图像的人脸超分辨率重构方法.模式识别与人工智能, 2020, 33(4): 325-336.
(FU L H, LU Z S, SUN X W, et al. Face Super-Resolution Reconstruction Method Fusing Reference Image. Pattern Recognition and Artificial Intelligence, 2020, 33(4): 325-336.)
[31] DOU H, CHEN C, HU X Y, et al. PCA-SRGAN: Incremental Orthogonal Projection Discrimination for Face Super-Resolution// Proc of the 28th ACM International Conference on Multimedia. New York, USA: ACM, 2020: 1891-1899.
[32] CHEN C F, GONG D H, WANG H, et al. Learning Spatial Attention for Face Super-Resolution. IEEE Transactions on Image Processing, 2021, 30: 1219-1231.
[33] VANDENHENDE S, GEORGOULIS S, VAN GANSBEKE W, et al. Multi-task Learning for Dense Prediction Tasks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3614-3633.
[34] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2022-03-15]. https://arxiv.org/pdf/1409.1556.pdf.
[35] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778.
[36] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269.
[37] 陈泓佑,陈帆,和红杰,等.基于样本特征解码约束的GANs.自动化学报, 2022, 48(9): 2288-2300.
(CHEN H Y, CHEN F, HE H J, et al. A GANs Model Based on Sample Feature Decoding Constraint. Acta Automatica Sinica, 2022, 48(9): 2288-2300.)
[38] MISRA I, SHRIVASTAVA A, GUPTA A, et al. Cross-Stitch Networks for Multi-task Learning// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 3394-4003.
[39] CIPOLLA R, GAL Y, KENDALL A.Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 7482-7491.
[40] CHEN H Y, HE H J, CHEN F.Multi-penalty Functions GANs via Multi-task Learning// Proc of the International Conference on Artificial Intelligence and Security. Berlin, Germany: Springer, 2021: 14-26.
[41] SONG H, KIM M, PARK D, et al. Learning from Noisy Labels with Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI: 10.1109/TNNLS.2022.3152527.
[42] GHOSH A, KUMAR H, SASTRY P S.Robust Loss Functions under Label Noise for Deep Neural Networks// Proc of the 31st AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI, 2017: 1919-1925.
[43] PATRINI G, ROZZA A, MENON A K, et al. Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2233-2241.
[44] CHEN H Y, HE H J, CHEN F, et al. Antinoise Learning and Coalitional Game GAN. International Journal of Pattern Recognition and Artificial Intelligence, 2022, 36(1). DOI: 10.1142/S0218001422520012.
[45] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[46] SHMELKOV K, SCHMID C, ALAHARI K.How Good Is My GAN?// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 218-234.