Abstract Since pixel level features in the data-driven face sketch synthesis algorithms lack robustness to illumination variation and complex background, the quality of synthesized face sketches is poor. In this paper, a robust face sketch synthesis algorithm based on deep probabilistic graphical models is proposed. A preprocessing method is adopted to adjust illumination brightness and face pose of an input photo to make them consistent with the training photos. Instead of pixel feature, deep feature representation is utilized for neighbor selecting. A deep probabilistic graphical model is employed to jointly model the weight of sketch reconstruction and the weight of deep feature, and therefore the best reconstruction representation of the synthetic image is obtained. A fast nearest neighbor search method is proposed to speed up sketch synthesis. Experimental results verify robustness and rapidity of the proposed algorithm.
Fund:Supported by National Natural Science Foundation of China(No.61876142)
Corresponding Authors:
WANG Nannan, Ph.D., professor. His research interests include computer vision, pattern recognition and machine learning.
About author:: ZHANG Yuqian, master student. Her research interests include computer vision, pa-ttern recognition and machine learning;GAO Fangyuan, undergraduate student.
[1] 王楠楠,李 洁,高新波.人脸画像合成研究的综述与对比分析.模式识别与人工智能, 2018, 31(1): 37-48. (WANG N N, LI J, GAO X B. A Review and Comparison Study on Face Sketch Synthesis. Pattern Recognition and Artificial Intelligence, 2018, 31(1): 37-48.) [2] 高新波,王楠楠,彭春蕾,等.基于三元空间融合的人脸图像模式识别.模式识别与人工智能, 2015, 28(9): 811-821. (GAO X B, WANG N N, PENG C L, et al. Facial Image Pattern Recognition Based on Triple Space Fusion. Pattern Recognition and Artificial Intelligence, 2015, 28(9): 811-821.). [3] WANG N N, TAO D C, GAO X B, et al. A Comprehensive Survey to Face Hallucination. International Journal of Computer Vision, 2014, 106(1): 9-30. [4] TANG X O, WANG X G. Face Photo Recognition Using Sketches // Proc of the International Conference on Image Processing. Washing- ton, USA: IEEE, 2002: 257-260. [5] LIU Q S, TANG X O, JIN H L, et al. A Nonlinear Approach for Face Sketch Synthesis and Recognition // Proc of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2005, I: 1005-1010. [6] SONG Y B, BAO L C, YANG Q X, et al. Real-Time Exemplar-Based Face Sketch Synthesis // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 800-813. [7] CHANG L, ZHOU M Q, HAN Y J, et al. Face Sketch Synthesis via Sparse Representation // Proc of the 20th International Conference on Pattern Recognition. Washington, USA: IEEE, 2010: 2146-2149. [8] GAO X B, WANG N N, TAO D C, et al. Face Sketch-Photo Synthesis and Retrieval Using Sparse Representation. IEEE Transactions on Circuits Systems for Video Technology, 2012, 22(8): 1213-1226. [9] WANG N N, GAO X B, LI J. Random Sampling for Fast Face Sketch Synthesis. Pattern Recognition, 2017, 76: 215-227. [10] GAO X B, ZHONG J J, LI J, et al. Face Sketch Synthesis Algorithm Based on E-HMM and Selective Ensemble. IEEE Transac-tions on Circuit Systems for Video Technology, 2008, 18(4): 487-496. [11] WANG X G, TANG X O. Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(11): 1955-1967. [12] ZHOU H, KUANG Z H, WONG K K. Markov Weight Fields for Face Sketch Synthesis // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2012: 1091-1097. [13] WANG N N, GAO X B, SUN L Y, et al. Bayesian Face Sketch Synthesis. IEEE Transactions on Image Processing, 2017, 26(3): 1264-1274. [14] ZHU M R, WANG N N. A Simple and Fast Method for Face Sketch Synthesis // Proc of the International Conference on Internet Multimedia Computing and Service. New York, USA: ACM, 2016: 168-171. [15] ZHANG L L, LIN L, WU X, et al. End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning // Proc of the 5th ACM International Conference on Multimedia Retrieval. New York, USA: ACM, 2015: 627-634. [16] WANG N N, ZHA W J, LI J, et al. Back Projection: An Effective Postprocessing Method for GAN-Based Face Sketch Synthesis. Pa-ttern Recognition Letters, 2018, 107: 59-65. [17] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1125-1134. [18] ZHU J Y, PARK T, ISOLA P, et al. Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 2242-2251. [19] ZHU M R, WANG N N, GAO X B, et al. Deep Graphical Feature Learning for Face Sketch Synthesis // Proc of the 26th International Joint Conference on Artificial Intelligence. New York, USA: ACM, 2017: 3574-3580. [20] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2019-04-25]. https://arxiv.org/pdf/1409.1556.pdf. [21] ZHANG Y Q, WANG N N, ZHANG S C, et al. Fast Face Sketch Synthesis via KD-Tree Search // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 64-77. [22] SONG Y B, ZHANG J W, BAO L C, et al. Fast Preprocessing for Robust Face Sketch Synthesis // Proc of the International Joint Conference on Artificial Intelligence. New York, USA: ACM, 2017: 4530-4536. [23] PIZER S M, AMBURN E P, AUSTIN J D, et al. Adaptive Histogram Equalization and Its Variations. Computer Vision, Graphics, and Image Processing, 1987, 39(3): 355-368. [24] KAZEMI V, SULLIVAN J. One Millisecond Face Alignment with an Ensemble of Regression Trees // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2014: 1867-1874. [25] MARTINEZ A, BENAVENTE R. The AR Face Database. Technical Report, 24. Columbus, USA: The Ohio State University, 1998. [26] MESSER K, MATAS J, KITTLER J, et al. XM2VTS: The Extended M2VTS Database // Proc of the 2nd International Confe-rence on Audio and Video-Based Biometric Person Authentication. Berlin, Germany: Springer, 1999: 72-77. [27] WANG Z, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. [28] SHEIKH H R, BOVIK A C. Image Information and Visual Quality. IEEE Transactions on Image Processing, 2006, 15(2): 430-444.
2019, Vol. 32 
