基于多任务对抗和抗噪对抗学习的人脸超分辨率算法

doi:10.16451/j.cnki.issn1003-6059.202210001

摘要
图/表
参考文献(0)
相关文章 (15)

全文: PDF (2122 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

高倍率单幅人脸图像超分辨率重建是一项具有实用价值但困难的任务.在人脸超分辨率任务中,端到端网络超分辨率图像较模糊,图像真实性和人眼视觉效果较差.针对上述问题,文中提出基于多任务对抗和抗噪对抗学习的人脸超分辨率算法.算法分为端到端网络学习阶段和网络参数微调阶段.为了提高端到端学习效果,设计深度多任务拉普拉斯金字塔网络,并结合多任务对抗学习.主任务为端到端学习,子任务为优化对抗学习惩罚项函数.为了改进通过对抗学习并微调主任务网络参数后的效果,在对抗学习的判别器优化过程中,融入抗噪对抗学习.实验表明,文中算法能使人脸超分辨率图像更具有图像真实性,更符合人眼视觉习惯.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	陈泓佑
	陈帆
	和红杰
	蒋桐雨

关键词 ：深度学习, 人脸超分辨率(FSR), 多任务对抗学习(MTAL), 抗噪对抗学习(ANAL), 多任务拉普拉斯金字塔网络(MTLapNet)

Abstract：

The super-resolution(SR) of high magnification single face image is a hard but valuable task. In the face super-resolution(FSR) task, the end-to-end network SR image is fuzzy, and the photoreality and human visual effect are poor. Aiming at the problems, a FSR algorithm based on multi-task adversarial learning(MTAL) and antinoise adversarial learning(ANAL) is proposed. The algorithm is divided into end-to-end network learning and network parameters fine-tuning. To improve the end-to-end learning result, a deep multi-task Laplacian pyramid network(MTLapNet) is designed and integrated with MTAL. The main task is end-to-end learning, while the subtask is the optimization of adversarial learning penalty function. To improve the result of adversarial learning and parameters fine-tuning of the main task network, ANAL is integrated into the optimization process of discriminator of adversarial learning. The experiments show that the proposed algorithm can make the FSR image more photo-realistic and more consistent with human visual habits.

Key words： Deep Learning Face Super-Resolution(FSR) Multi-task Adversarial Learning(MTAL) Antinoise Adversarial Learning(ANAL) Multi-task Laplacian Pyramid Network(MTLapNet)

收稿日期: 2022-04-18

ZTFLH:	TP 391.41
	TP 183

基金资助:

国家自然科学基金项目(No.U1936113,61872303)资助

通讯作者: 和红杰,博士,教授,主要研究方向为图像取证、图像处理.E-mail:hjhe@home.swjtu.edu.cn.

作者简介: 陈泓佑,博士,主要研究方向为机器学习、图像处理.E-mail:chy2019@foxmail.com. 陈帆,博士,副教授,主要研究方向为多媒体安全、计算机应用.E-mail:fchen@home.swjtu.edu.cn. 蒋桐雨,硕士研究生,主要研究方向为人脸超分辨率.E-mail:jiangty971018@163.com.

引用本文:

陈泓佑, 陈帆, 和红杰, 蒋桐雨. 基于多任务对抗和抗噪对抗学习的人脸超分辨率算法[J]. 模式识别与人工智能, 2022, 35(10): 863-880. CHEN Hongyou, CHEN Fan, HE Hongjie, JIANG Tongyu. Face Super-Resolution Algorithm Based on Multi-task Adversarial and Antinoise Adversarial Learning. Pattern Recognition and Artificial Intelligence, 2022, 35(10): 863-880.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202210001 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2022/V35/I10/863

[1] CHEN H G, HE X H, QING L B, et al. Real-World Single Image Super-Resolution: A Brief Review. Information Fusion, 2022, 79: 124-145.
[2] ZHANG M J, XIN J W, ZHANG J, et al. Curvature Consistent Network for Microscope Chip Image Super-Resolution. IEEE Transactions on Neural Network and Learning Systems, 2022. DOI: 10.1109/TNNLS.2022.3168540.
[3] DONG C, LOY C C, HE K M, et al. Image Super-Resolution Using Deep Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[4] KIM J, LEE J K, LEE K M.Accurate Image Super-Resolution Using Very Deep Convolutional Networks// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 1646-1654.
[5] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5835-5843.
[6] LIM B, SON S, KIM H, et al. Enhanced Deep Residual Networks for Single Image Super-Resolution// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1132-1140.
[7] LI Z, YANG J L, LIU Z, et al. Feedback Network for Image Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 3862-3871.
[8] ZHAO X L, LIAO Y, HE T, et al. FC²N: Fully Channel-Concatenated Network for Single Image Super-Resolution[C/OL].[2021-03-15]. https://arxiv.org/pdf/1907.03221.pdf.
[9] LIU J, ZHANG W J, TANG Y T, et al. Residual Feature Aggregation Network for Image Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2356-2365.
[10] 张铭津,彭晓琪,郭杰,等.基于多残差网络的结构保持超分辨重建.模式识别与人工智能, 2021, 34(3): 232-240.
(ZHANG M J, PENG X Q, GUO J, et al. Structure-Preserving Super-Resolution Reconstruction Based on Multi-residual Network. Pattern Recognition and Artificial Intelligence, 2021, 34(3): 232-240.)
[11] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: Image Restoration Using Swin Transformer// Proc of the IEEE/CVF Internatio-nal Conference on Computer Vision Workshops. Washington, USA: IEEE, 2021: 1833-1844.
[12] ZHANG M J, WU Q Q, ZHANG J, et al. Fluid Micelle Network for Image Super-Resolution Reconstruction. IEEE Transactions on Cybernetics, 2022. DOI: 10.1109/TCYB.2022.3163294.
[13] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al.Ge-nerative Adversarial Nets// Proc of the 27th International Confe-rence on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2014: 2672-2680.
[14] HONG Y J, HWANG U, YOO J, et al. How Generative Adversa-rial Networks and Their Variants Work: An Overview. ACM Computing Surveys, 2019, 52(1). DOI: 10.1145/3301282.
[15] LEDIG C, THEIS L, HUSZÁR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network// Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 105-114.
[16] WANG X T, YU K, WU S X, et al. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks// Proc of the Euro-pean Conference on Computer Vision. Berlin, Germany: Springer, 2018: 63-79.
[17] RAKOTONIRINA N C, RASOANAIVO A.ESRGAN+: Further Improving Enhanced Super-Resolution Generative Adversarial Network// Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington, USA: IEEE, 2020: 3637-3641.
[18] HARIS M, SHAKHNAROVICH G, UKITA N.Deep Back-Projection Networks for Super-Resolution// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 1664-1673.
[19] HUI Z, LI J, GAO X B, et al. Progressive Perception-Oriented Network for Single Image Super-Resolution. Information Sciences, 2021, 546: 769-786.
[20] JIANG J J, WANG C Y, LIU X M, et al. Deep Learning-Based Face Super-Resolution: A Survey. ACM Computing Surveys, 2003, 55(1). DOI: 10.1145/3485132.
[21] LIU H, ZHENG X Y, HAN J G, et al. Survey on GAN-Based Face Hallucination with Its Model Development. IET Image Processing, 2019, 13(14): 2662-2672.
[22] GOSWAMI S, AAKANKSH A, RAJAGOPALAN A N. Robust Super-Resolution of Real Faces Using Smooth Features// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 169-185.
[23] LI X M, CHEN C F, ZHOU S C, et al. Blind Face Restoration via Deep Multi-scale Component Dictionaries// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 399-415.
[24] CHEN Y, TAI Y, LIU X M, et al. FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 2492-2501.
[25] MA C, JIANG Z Y, RAO Y M, et al. Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation// Proc of the IEEE/CVF Conference on Compu-ter Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 5568-5577.
[26] ZHANG Y C, WU Y, CHEN L.MSFSR: A Multi-stage Face Super-Resolution with Accurate Facial Representation via Enhanced Facial Boundaries// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 2120-2129.
[27] 蒋桐雨,陈帆,和红杰.基于非对称U型金字塔重建的轻量级人脸超分辨率网络.山东大学学报 (工学版), 2022, 52(1): 1-8, 18.
(JIANG T Y, CHEN F, HE H J.Lightweight Face Super-Resolution Network Based on Asymmetric U-Pyramid Reconstruction. Journal of Shandong University (Engineering Science), 2022, 52(1): 1-8, 18.)
[28] KIM D, KIM M, KWON G, et al. Progressive Face Super-Resolution via Attention to Facial Landmark[C/OL].[2022-03-15]. https://arxiv.org/pdf/1908.08239v1.pdf.
[29] BULAT A, TZIMIROPOULOS G.Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 109-117.
[30] 付利华,卢中山,孙晓威,等.融合参考图像的人脸超分辨率重构方法.模式识别与人工智能, 2020, 33(4): 325-336.
(FU L H, LU Z S, SUN X W, et al. Face Super-Resolution Reconstruction Method Fusing Reference Image. Pattern Recognition and Artificial Intelligence, 2020, 33(4): 325-336.)
[31] DOU H, CHEN C, HU X Y, et al. PCA-SRGAN: Incremental Orthogonal Projection Discrimination for Face Super-Resolution// Proc of the 28th ACM International Conference on Multimedia. New York, USA: ACM, 2020: 1891-1899.
[32] CHEN C F, GONG D H, WANG H, et al. Learning Spatial Attention for Face Super-Resolution. IEEE Transactions on Image Processing, 2021, 30: 1219-1231.
[33] VANDENHENDE S, GEORGOULIS S, VAN GANSBEKE W, et al. Multi-task Learning for Dense Prediction Tasks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7): 3614-3633.
[34] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2022-03-15]. https://arxiv.org/pdf/1409.1556.pdf.
[35] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778.
[36] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269.
[37] 陈泓佑,陈帆,和红杰,等.基于样本特征解码约束的GANs.自动化学报, 2022, 48(9): 2288-2300.
(CHEN H Y, CHEN F, HE H J, et al. A GANs Model Based on Sample Feature Decoding Constraint. Acta Automatica Sinica, 2022, 48(9): 2288-2300.)
[38] MISRA I, SHRIVASTAVA A, GUPTA A, et al. Cross-Stitch Networks for Multi-task Learning// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 3394-4003.
[39] CIPOLLA R, GAL Y, KENDALL A.Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics// Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 7482-7491.
[40] CHEN H Y, HE H J, CHEN F.Multi-penalty Functions GANs via Multi-task Learning// Proc of the International Conference on Artificial Intelligence and Security. Berlin, Germany: Springer, 2021: 14-26.
[41] SONG H, KIM M, PARK D, et al. Learning from Noisy Labels with Deep Neural Networks: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 2022. DOI: 10.1109/TNNLS.2022.3152527.
[42] GHOSH A, KUMAR H, SASTRY P S.Robust Loss Functions under Label Noise for Deep Neural Networks// Proc of the 31st AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI, 2017: 1919-1925.
[43] PATRINI G, ROZZA A, MENON A K, et al. Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach// Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2233-2241.
[44] CHEN H Y, HE H J, CHEN F, et al. Antinoise Learning and Coalitional Game GAN. International Journal of Pattern Recognition and Artificial Intelligence, 2022, 36(1). DOI: 10.1142/S0218001422520012.
[45] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[46] SHMELKOV K, SCHMID C, ALAHARI K.How Good Is My GAN?// Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 218-234.