Chinese Portrait Painting Style Transfer Algorithm
SHENG Jiachuan1, DONG Yufan1, LI Xiaomei2, LI Yuzhi1
1. School of Science and Technology, Tianjin University of Finance and Economics, Tianjin 300222 2. Department of Sci-tech Innovation and Achievement Transformation, Tianjin University of Finance and Economics, Tianjin 300222
Abstract Style transfer algorithms can generate target artworks quickly. However, the problems are caused by applying style transfer algorithms directly to Chinese paintings, like uneven feature distribution and inconsistent face recognition. To address these issues, a Chinese portrait painting style transfer algorithm based on convolutional neural network(CNN) is proposed. Firstly, a brushstroke control restriction is proposed to guide the texture distribution of the image for freehand brushwork and fine brushwork of Chinese portrait painting. Then, Chinese painting moving distance is proposed to measure content and style features and transfer the style of Chinese painting to portrait photos harmoniously. Finally, the restriction for improving the loss network is put forward based on the ink tone characteristics and the blank space reservation. Experiments show that the proposed algorithm is superior in Chinese painting style and the results maintain the consistency of face recognition.
Fund:National Natural Science Foundation of China(No.61502331), MOE Project of Humanities and Social Science(No.18YJA630057,19YJA630046), Natural Science Foundation of Tianjin(No.18JCYBJC85100)
Corresponding Authors:
LI Yuzhi, master, lecturer. Her research interests include machine learning and multimedia processing.
About author:: SHENG Jiachuan, Ph.D., professor. Her research interests include digital multimedia processing and pattern recognition. DONG Yufan, master student. Her research interests include computer vision. LI Xiaomei, master, professor. Her research interests include science and technology innovation and management.
[1] JING Y C, YANG Y Z, FENG Z L, et al. Neural Style Transfer: A Review. IEEE Transactions on Visualization and Computer Gra-phics, 2020, 26(11): 3365-3385. [2] WU R Z, GU X D, TAO X, et al. Landmark Assisted CycleGAN for Cartoon Face Generation[C/OL]. [2020-07-01]. https://arxiv.org/pdf/1907.01424v1.pdf. [3] HE B, GAO F, MA D Q, et al. ChipGAN: A Generative Adversa-rial Network for Chinese Ink Wash Painting Style Transfer // Proc of the 26th ACM International Conference on Multimedia. New York, USA: ACM, 2018: 1172-1180. [4] GATYS L A, ECKER A S, BETHGE M. Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2414-2423. [5] WILMOT P, RISSER E, BARNES C. Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses[C/OL]. [2020-07-01]. https://arxiv.org/pdf/1701.08893v1.pdf. [6] LI S H, XU X X, NIE L Q, et al. Laplacian-Steered Neural Style Transfer // Proc of the 25th ACM International Conference on Multimedia. New York, USA: ACM, 2017: 1716-1724. [7] MECHREZ R, TALMI I, ZELNIK-MANORA L. The Contextual Loss for Image Transformation with Non-aligned Data // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 800-815. [8] HUANG X, BELONGIE S. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 1510-1519. [9] JING Y C, LIU Y, YANG Y Z, et al. Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 244-260. [10] KOLKIN N, SALAVON J, SHAKHNAROCIVH G. Style Transfer by Relaxed Optimal Transport and Self-Similarity // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 10043-10052. [11] KHAN A, AHMAD M, NAQVI N, et al. Photographic Painting Style Transfer Using Convolutional Neural Networks. Multimedia Tools and Applications, 2019, 78(14): 19565-19586. [12] KIM S S Y, KOLKIN N, SALAVON J, et al. Deformable Style Transfer // Proc of the European Conference on Computer Vision[C/OL]. [2020-07-01]. https://arxiv.org/pdf/2003.11038.pdf. [13] LIAO J, YAO Y, YUAN L, et al. Visual Attribute Transfer through Deep Image Analogy[C/OL]. [2020-07-01]. https://arxiv.org/pdf/1705.01088.pdf. [14] YAO Y, REN J Q, XIE X S, et al. Attention-Aware Multi-stroke Style Transfer // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1467-1475. [15] 施荣晓,叶东毅,陈昭炯.梯度控制与鉴别特征引导的写实类人物肖像插画转换方法.模式识别与人工智能, 2020, 33(11): 959-971. (SHI R X, YE D Y, CHEN Z J. Gradient Controlled and Discri-minative Features Guided Image-to-Image Translation Method Towards Realistic Portrait Illustrations. Pattern Recognition and Artificial Intelligence, 2020, 33(11): 959-971.) [16] SHENG J C, JIANG J M. Recognition of Chinese Artists via Windowed and Entropy Balanced Fusion in Classification of Their Authored Ink and Wash Paintings(IWPs). Pattern Recognition, 2014, 47(2): 612-622. [17] 盛家川,李玉芝.国画的艺术目标分割及深度学习与分类.中国图象图形学报, 2018, 23(8): 1193-1206. (SHENG J C, LI Y Z. Learning Artistic Objects for Improved Classification of Chinese Paintings. Journal of Image and Graphics, 2018, 23(8): 1193-1206.) [18] 李玉芝,盛家川,华 斌.中国画分类的改进嵌入式学习算法.计算机辅助设计与图形学学报, 2018, 30(5): 893-900. (LI Y Z, SHENG J C, HUA B. Improved Embedded Learning for Classification of Chinese Paintings. Journal of Computer-Aided Design and Computer Graphics, 2018, 30(5): 893-900.) [19] 盛家川,陈雅琦,王 君,等.融合人类认知网络优化的中国画情感识别.模式识别与人工智能, 2020, 33(2): 141-149. (SHENG J C, CHEN Y Q, WANG J, et al. Chinese Paintings Sentiment Recognition via CNN Optimization with Human Cognition. Pattern Recognition and Artificial Intelligence, 2020, 33(2): 141-149.) [20] 盛家川,陈雅琦,韩亚洪.深层网络特征聚合重标定的中国画情感分类算法.计算机辅助设计与图形学学报, 2020, 32(9): 1420-1429. (SHENG J C, CHEN Y Q, HAN Y H. Sentiment Classification of Chinese Paintings via Feature Recalibration of Deep Network Aggregation. Journal of Computer-Aided Design and Computer Graphics, 2020, 32(9): 1420-1429.) [21] 顾 杨,陈昭炯,陈 灿,等.基于CGAN的中国山水画布局可调的仿真生成方法.模式识别与人工智能, 2019, 32(9): 844-854. (GU Y, CHEN Z J, CHEN C, et al. Layout Adjustable Simulated Generation Method for Chinese Landscape Paintings Based on CGAN. Pattern Recognition and Artificial Intelligence, 2019, 32(9): 844-854.) [22] SHENG J C, SONG C F, WANG J, et al. Convolutional Neural Network Style Transfer towards Chinese Paintings. IEEE Access, 2019, 7: 163719-163728. [23] LI B, XIONG C M, WU T F, et al. Neural Abstract Style Transfer for Chinese Traditional Painting // Proc of the Asian Conference on Computer Vision. Berlin, Germany: Springer, 2018: 212-227. [24] HARIHARAN B, ARBELAEZ P, GIRSHICK R, et al. Hypercolumns for Object Segmentation and Fine-Grained Localization // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2015: 447-456. [25] 景年昭,杨 维.基于RCF的精细边缘检测模型.计算机应用, 2019, 39(9): 2535-2540. (JING N Z, YANG W. Fine Edge Detection Model Based on RCF. Journal of Computer Applications, 2019, 39(9): 2535-2540.) [26] YU F, KOLTUN V. Multi-scale Context Aggregation by Dilated Convolutions[C/OL]. [2020-07-01]. https://arxiv.org/pdf/1511.07122v3.pdf. [27] RUBNER Y, TOMASI C, GUIBAS L J. The Earth Mover's Distance as a Metric for Image Retrieval. International Journal of Computer Vision, 2000, 40(2): 99-121. [28] SHEN X Y, TAO X, GAO H Y, et al. Deep Automatic Portrait Matting // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 92-107. [29] LUAN F J, PARIS S, SHECHTMAN E, et al. Deep Photo Style Transfer // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6997-7005. [30] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the Inception Architecture for Computer Vision // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2818-2826.
2021, Vol. 34 
