Face Sketch-Photo Synthesis Method Based on Multi-residual Dynamic Fusion Generative Adversarial Networks
SUN Rui1,2, SUN Qijing1,2, SHAN Xiaoquan1,2, ZHANG Xudong1
1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601; 2. Anhui Key Laboratory of Industry Safety and Emergency Technology, Hefei University of Technology, Hefei 230009
Abstract Aiming at the low definition and blurry details in the current face sketch-photo synthesis methods, a face sketch-photo synthesis method based on multi-residual dynamic fusion generative adversarial network is proposed. Firstly, a multi-residual dynamic fusion network is designed. Features are extracted from different dense residual modules and then the residual learning is conducted. Then, the corresponding offsets are generated on the basis of the diverse residual features at different levels. The sampling coordinates of the convolution kernels in different locations are changed according to the offsets. Consequently, the network is focused on important feature adaptively, and geometric detail information and high-level semantic information are integrated effectively without gradual information dropping and redundant information interference. Moreover, a multi-scale perceptual loss is introduced to conduct perceptual comparison on the synthetic images of different resolutions for the regularization of synthetic images from coarse to fine. Experiments on Chinese University of Hong Kong face sketch dataset show that the proposed method produces high-definition images with full detail and consistent color and the synthesized image is closer to real face images.
Fund:General Project of National Natural Science Foundation of China(No.6147115461876057), Key Research and Development Plan of Anhui Province(No.202004D07020012)
Corresponding Authors:SUN Rui, Ph.D., professor. His research interests include computer vision and machine learning.
About author:: SUN Qijing, master student. His research interests include image information processing and computer vision. SHAN Xiaoquan, master student. His research interests include image information processing and computer vision. ZHANG Xudong, Ph.D., professor. His research interests include intelligent information processing and machine vision.
SUN Rui,SUN Qijing,SHAN Xiaoquan等. Face Sketch-Photo Synthesis Method Based on Multi-residual Dynamic Fusion Generative Adversarial Networks[J]. Pattern Recognition and Artificial Intelligence, 2022, 35(3): 207-222.
[1] LIU Q S, TANG X O, JIN H L, ,et al. A Nonlinear Approach for Face Sketch Synthesis and Recognition // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2005, I: 1005-1010. [2] WANG X G, TANG X O.Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(11): 1955-1967. [3] CHANG L, ZHOU M Q, DENG X M, et al. Face Sketch Synthesis via Multivariate Output Regression // Proc of the 14th International Conference on Human-Computer Interaction: Design and Development Approaches. Berlin,Germany: Springer, 2011: 555-561. [4] ZHANG S C, GAO X B, WANG N N, et al. Face Sketch Synthesis via Sparse Representation-Based Greedy Search. IEEE Transactions on Image Processing, 2015, 24(8): 2466-2477. [5] ZHANG D Y, LIN L, CHEN T S, et al. Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning. IEEE Transactions on Image Processing, 2017, 26(1): 328-339. [6] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Ge-nerative Adversarial Nets // Proc of the 27th International Confe-rence on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014, II: 2672-2680. [7] KINGMA D P, WELLING M. Auto-Encoding Variational Bayes[C/OL]. [2021-08-23]. https://arxiv.org/pdf/1312.611.pdf. [8] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2017: 5967-5976. [9] WANG T C, LIU M Y, ZHU J Y, et al. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington,USA: IEEE, 2018: 8798-8807. [10] ZHU J Y, PARK T, ISOLA P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks // Proc of the IEEE International Conference on Computer Vision. Washington,USA: IEEE, 2017: 2242-2251. [11] YI Z L, ZHANG H, TAN P, et al. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation // Proc of the IEEE International Conference on Computer Vision. Washington,USA: IEEE, 2017: 2868-2876. [12] LU Y Y, WU S Z, TAI Y W, et al. Image Generation from Sketch Constraint Using Contextual GAN // Proc of the European Confe-rence on Computer Vision. Berlin,Germany: Springer, 2018: 213-228. [13] WANG N N, ZHA W J, LI J, et al. Back Projection: An Effective Postprocessing Method for GAN-Based Face Sketch Synthesis. Pa-ttern Recognition Letters, 2018, 107: 59-65. [14] WANG L D, SINDAGI V, PATEL V.High-Quality Facial Photo-Sketch Synthesis Using Multi-adversarial Networks // Proc of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Washington,USA: IEEE, 2018: 83-90. [15] CHEN C F, LIU W, TAN X, et al. Semi-Supervised Learning for Face Sketch Synthesis in the Wild // Proc of the Asian Conference on Computer Vision. Berlin,Germany: Springer, 2018: 216-231. [16] ZHENG J Y, SONG W R, WU Y H, et al. Feature Encoder Guided Generative Adversarial Network for Face Photo-Sketch Synthesis. IEEE Access, 2019, 7: 154971-154985. [17] BABU K K, DUBEY S R.CSGAN: Cyclic-Synthesized Generative Adversarial Networks for Image-to-Image Transformation. Expert Systems with Applications, 2021, 169. DOI:10.1016/j.eswa.2020.114431. [18] BABU K K, DUBEY S R.CDGAN: Cyclic Discriminative Generative Adversarial Networks for Image-to-Image Transformation[C/OL]. [2021-08-23].https://arxiv.org/pdf/2001.05489.pdf. [19] HAN J L, SHOEIBY M, PETERSSON L, et al. Dual Contrastive Learning for Unsupervised Image-to-Image Translation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington,USA: IEEE, 2021: 746-755. [20] YI R, LIU Y J, LAI Y K, et al. APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington,USA: IEEE, 2019: 10735-10744. [21] KIM J, KIM M, KANG H, et al. U-GAT-IT: Unsupervised Ge-nerative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation[C/OL].[2021-08-23]. https://arxiv.org/pdf/1907.10830v3.pdf. [22] ZHU M R, LI J, WANG N N, et al. Learning Deep Patch Representation for Probabilistic Graphical Model-Based Face Sketch Synthesis. International Journal of Computer Vision, 2021, 129(6): 1820-1836. [23] ZHU M R, LI J, WANG N N, et al. Knowledge Distillation for Face Photo-Sketch Synthesis. IEEE Transactions on Neural Net-works and Learning Systems, 2020. DOI: 10.1109/TNNLS.2020.3030536. [24] ZHU M R, LI J, WANG N N, et al. A Deep Collaborative Framework for Face Photo-Sketch Synthesis. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(10): 3096-3108. [25] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-08-23]. https://arxiv.org/pdf/1409.1556.pdf. [26] JOHNSON J, ALAHI A, LI F F.Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Con-ference on Computer Vision. Berlin,Germany: Springer, 2016: 694-711. [27] GATYS L A, ECKER A S, BETHGE M.Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2016: 2414-2423. [28] MAO X D, LI Q, XIE H R, et al. Least Squares Generative Adversarial Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2813-2821. [29] ZHANG W, WANG X G, TANG X O.Coupled Information-Theoretic Encoding for Face Photo-Sketch Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2011: 513-520. [30] KINGMA D P, BA J. Adam: A Method for Stochastic Optimization[C/OL]. [2021-08-23]. https://arxiv.org/pdf/1412.6980.pdf.
2022, Vol. 35 
