Abstract Face sketch synthesis aims to transform a face photo into a face sketch. For existing methods, the generated sketches are over-smooth and the pre-training on additional large scale datasets is required. In this paper, a deep bidirectional network based on the least mean square error reconstruction(Lmser) self-organizing network is constructed with a feature of duality in paired neurons(DPN) to generate face sketch. DPN is realized with bidirectional shortcuts between encoder and decoder. It helps transfer features learn from different layers of the Lmser to improve texture details in synthesized sketch. Another sketch-to-photo mapping network is built by a complement Lmser with converse direction sharing the same structure. The bidirectional mappings form an outer Lmser network with DPN enforce consistency between the paired blocks in a global manner. Experiments on benchmark datasets demonstrate that the performance of the proposed method is superior, and it is more applicable and does not need pre-training on additional datasets.
Fund:Supported by the National Key Research and Development Program of the Ministry of Science and Technology of China(No.2018AAA0100700), Shanghai Municipal Science and Technology Major Project(No.2021SHZDZX0102)
Corresponding Authors:XÜ Lei, Ph.D., professor. His research interests include machine learning, casual computation, bidirectional intelligence, AlphaGo-like algorithm, computational health and computational intelligence in finance.
About author:: About Author:SHENG Qingjie, master student. His research interests include face sketch synthesis and deep bidirectional learning.
SU Ruidan, Ph.D., assistant professor. His research interests include machine lear-ning and bidirectional intelligence.
Tu Shikui, Ph.D., associate professor. His research interests include machine lear-ning and bioinformatics.
[1] WANG X G, TANG X O. Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 2009, 31(11): 1955-1967. [2] SONG Y B, BAO L C, YANG Q X, et al. Real-Time Exemplar-Based Face Sketch Synthesis // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 800-813. [3] WANG N N, GAO X B, LI J. Random Sampling for Fast Face Sketch Synthesis. Pattern Recognition, 2018, 76: 215-227. [4] ZHANG L L, LIN L, WU X, et al. End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning // Proc of the 5th ACM International Conference on Multimedia Retrieval. New York, USA: ACM, 2015: 627-634. [5] WANG N N, ZHA W J, LI J, et al. Back Projection: An Effective Postprocessing Method for GAN-Based Face Sketch Synthesis. Pattern Recognition Letters, 2017, 107: 59-65. [6] ZHANG D Y, LIN L, CHEN T S, et al. Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning. IEEE Transactions on Image Processing, 2017, 26(1): 328-339. [7] YU J, XU X X, GAO F, et al. Toward Realistic Face Photo-Sketch Synthesis via Composition-Aided GANs. IEEE Transactions on Cybernetics, 2021, 51(9): 4350-4362. [8] TANG X O, WANG X G. Face Sketch Synthesis and Recognition // Proc of the 9th IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2003, I: 687-694. [9] LIU Q S, TANG X O, JIN H L, et al. A Nonlinear Approach for Face Sketch Synthesis and Recognition // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2005: 1005-1010. [10] ZHU M K, LI J, WANG N N, et al. Learning Deep Patch Representation for Probabilistic Graphical Model-Based Face Sketch Synthesis. International Journal of Computer Vision, 2021, 129(6): 1820-1836. [11] ZHU M R, LI J, WANG N N, et al. Knowledge Distillation for Face Photo-Sketch Synthesis. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 893-906. [12] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative Adversarial Nets // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014, II: 2672-2680. [13] MAKHZANI A, JAITLY N S N, GOODFELLOW I, et al. Adversarial Autoencoders[J/OL].[2022-03-25]. https://arxiv.org/pdf/1511.05644.pdf. [14] LIU M Y, TUZEL O.Coupled Generative Adversarial Networks // Proc of the 30th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2016: 469-477. [15] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5967-5976. [16] CHEN D D, YUAN L, LIAO J, et al. StyleBank: An Explicit Representation for Neural Image Style Transfer // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2770-2779. [17] ZHU J Y, PARK T, ISOLA P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2242-2251. [18] WAN W G, YANG Y, LEE H J. Generative Adversarial Learning for Detail-Preserving Face Sketch Synthesis. Neurocomputing, 2021, 438: 107-121. [19] ZHANG M J, LI Y S, WANG N N, et al. Cascaded Face Sketch Synthesis under Various Illuminations. IEEE Transactions on Image Processing, 2020, 29: 1507-1521. [20] LI P, SHENG B, CHEN C L P. Face Sketch Synthesis Using Re-gularized Broad Learning System. IEEE Transactions on Neural Networks and Learning Systems, 2021. DOI: 10.1109/TNNLS.203070463 [21] ZHANG M J, LI J, WANG N N, et al. Compositional Model-Based Sketch Generator in Facial Entertainment. IEEE Transactions on Systems, Man, and Cybernetics, 2018, 48(3): 904-915. [22] JOHSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711. [23] XU L. Least Mean Square Error Reconstruction Principle for Self-Organizing Neural-Nets. Neural Networks, 1993, 6(5): 627-648. [24] BOURLARD H, KAMP Y. Auto-Association by Multilayer Perceptrons and Singular Value Decomposition. Biological Cybernetics, 1988, 59(4): 291-294. [25] XU L. An Overview and Perspectives on Bidirectional Intelligence: Lmser Duality, Double IA Harmony, and Causal Computation. IEEE/CAA Journal of Automatica Sinica, 2019, 6(4): 865-893. [26] HUANG W J, TU S K, XU L. Revisit Lmser from a Deep Learning Perspective // Proc of the International Conference on Intelligence Science and Big Data Engineering. Berlin, Germany: Springer, 2019: 197-208. [27] HUANG W J, TU S K, XU L. Deep CNN Based Lmser and Strengths of Two Built-In Dualities. Neural Processing Letters, 2020. DOI: 10.1007/s11063-020-10341-5. [28] LI P Y, TU S K, XU L. GAN Flexible Lmser for Super-Resolution // Proc of the 27th ACM International Conference on Multimedia. New York, USA: ACM, 2019: 756-764. [29] GUO Y Z, HUANG W J, CHEN Y J, et al. Regularize Network Skip Connections by Gating Mechanisms for Electron Microscopy Image Segmentation // Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2019: 868-873. [30] CAO B H, TU S K, XU L. Flexible-CLmser: Regularized Feedback Connections for Biomedical Image Segmentation // Proc of the IEEE International Conference on Bioinformatics and Biomedicine. Washington, USA: IEEE, 2021: 829-835. [31] MIRZA M, OSINDERO S. Conditional Generative Adversarial Nets[C/OL]. [2022-03-25].https://arxiv.org/pdf/1411.1784v1.pdf. [32] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [33] MAO X D, LI Q, XIE H R, et al. Least Squares Generative Adversarial Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2813-2821. [34] ZHANG W, WANG X G, TANG X O. Coupled Information-Theoretic Encoding for Face Photo-Sketch Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2011: 513-520. [35] TANG X O, WANG X G. Face Photo Recognition Using Sketch // Proc of the International Conference on Image Processing. Washington, USA: IEEE, 2002. DOI: 10.1109/ICIP.2002.1038008. [36] MARTINEZ A, BENAVENTE R. The AR Face Database. CVC Technical Report, 24. Barcelona, Spain: Autonomous University of Barcelona, 1998. [37] MESSER K, MATAS J, KITTLER J, et al.. XM2VTSDB: The Extended M2VTS Database[C/OL]. [2022-03-25]. http://www.ee.surrey.ac.uk/CVSSP/xm2vtsdb/docs/messer-avbpa99.pdf. [38] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2017: 6629-6640. [39] ZHANG L, ZHANG L, MOU X Q, et al. FSIM: A Feature Similarity Index for Image Quality Assessment. IEEE Transactions on Image Processing, 2011, 20(8): 2378-2386. [40] ZHU M R, WANG N N, GAO X B, et al. Deep Graphical Feature Learning for Face Sketch Synthesis // Proc of the 26th International Joint Conference on Artificial Intelligence. San Francisco, USA: IJCAI, 2017: 3574-3580.
2022, Vol. 35 
