Layout Adjustable Simulated Generation Method for Chinese Landscape Paintings Based on CGAN
GU Yang1,2, CHEN Zhaojiong1,2, CHEN Can1,2, YE Dongyi1,2
1.College of Mathematics and Computer Science, Fuzhou University, Fuzhou 350116 2.Key Laboratory of Spatial Data Mining and Information Sharing, Ministry of Education, Fuzhou University, Fuzhou 350116
Abstract：Creating a complete landscape painting via computer simulation is difficult without studying from global layout viewpoint. To address this issue, a layout-guided Chinese landscape painting simulation method for a complete painting generation is proposed. The characteristics of landscape paintings are taken into account in the design of feasible structures of layout label maps. Composition forms and elements of landscape paintings can be depicted using those structures. On the basis of condition generative adversarial network (CGAN) approach, a multi-scale feature fusion CGAN (MSFF-CGAN) is designed based on layouts and touches of landscape paintings. The proposed network is trained to accomplish heterogeneous transfer from a layout label map to a simulated landscape painting. To deal with rare availability of layout label maps for network training, a color pixel clustering algorithm with semantic correlation is used. In order to enhance the artistic reality of the generated landscape painting, a super resolution network named MemNet is incorporated to refine the texture details. Experimental results show that the proposed method is superior to existing methods in both integrity and artistic reality. Moreover, the proposed method can be used to handle simple graffiti sketches and modify simulated landscape paintings by editing label maps.
[1] LEE J. Simulating Oriental Black-Ink Painting. IEEE Computer Graphics and Applications, 1999, 19(3): 74-81. [2] WAY D L, SHIH Z C. The Synthesis of Rock Textures in Chinese Landscape Painting. Computer Graphics Forum, 2010, 20(3): 123-131. [3] 徐添辰,吴恩华.动态模拟中国水墨画中的笔画绘制.计算机辅助设计与图形学学报, 2016, 28(5): 742-749. (XU T C, WU E H. Animating Strokes in Drawing Process of Chinese Ink Painting. Journal of Computer-Aided Design and Computer Graphics, 2016, 28(5): 742-749.) [4] TANG F, DONG W M, MENG Y P, et al. Animated Construction of Chinese Brush Paintings. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(12): 3019-3031. [5] YU J H, LUO G M, PENG Q S. Image-Based Synthesis of Chinese Landscape Painting. Journal of Computer Science and Technology, 2003, 18(1): 22-28. [6] 张海嵩,尹小勤,于金辉.实时绘制3D中国画效果.计算机辅助设计与图形学学报, 2004, 16(11): 1485-1489. (ZHANG H S, YIN X Q, YU J H. Real-Time Rendering of 3D Chinese Painting Effects. Journal of Computer-Aided Design and Computer Graphics, 2004, 16(11): 1485-1489.) [7] WANG Z, SUN M J, SUN J Z, et al. Neural Network-Based Chinese Ink-Painting Art Style Learning // Proc of the IEEE International Conference on Intelligent Computing and Intelligent Systems. Washington, USA: IEEE, 2010: 462-466. [8] GATYS L A, ECKER A S, BETHGE M. Image Style Transfer Using Convolutional Neural Networks // Proc of the IEEE Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2414-2423. [9] WANG T C, LIU M Y, ZHU J Y, et al. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2018: 8798-8807. [10] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5967-5976. [11] PARK T, LIU M Y, WANG T C, et al. Semantic Image Synthesis with Spatially-Adaptive Normalization[C/OL]. [2019-03-21]. https://arxiv.org/pdf/1903.07291.pdf. [12] TAI Y, YANG J, LIU X M, et al. MemNet: A Persistent Memory Networks for Image Restoration // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4549-4557. [13] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Gene-rative Adversarial Networks // Proc of the 27th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2014: 2672-2680. [14] 王坤峰,苟 超,段艳杰,等.生成式对抗网络GAN的研究进展与展望.自动化学报, 2017, 43(3): 321-332. (WANG K F, GOU C, DUAN Y J. Generative Adversarial Networks: The State of the Art and Beyond. Acta Automatica Sinica, 2017, 43(3): 321-332.) [15] 蔡雨婷,陈昭炯,叶东毅.基于双层级联GAN的草图到真实感图像的异质转换.模式识别与人工智能, 2018, 31(10): 877-886. (CAI Y T, CHEN Z J, YE D Y. Bi-level Cascading GAN-Based Heterogeneous Conversion of Sketch-to-Realistic Images. Pattern Recognition and Artificial Intelligence, 2018, 31(10): 877-886.) [16] 唐贤伦,杜一铭,刘雨微,等.基于条件深度卷积生成对抗网络的图像识别方法.自动化学报, 2018, 44(5): 855-864. (TANG X L, DU Y M, LIU Y W, et al. Image Recognition with Conditional Deep Convolutional Generative Adversarial Networks. Acta Automatica Sinica, 2018, 44(5): 855-864.) [17] JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711. [18] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context Encoders: Feature Learning by Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2536-2544. [19] YOO D, KIM N, PARK S, et al. Pixel-Level Domain Transfer // Proc of the European Conference on Computer Vision. Berlin, Ger- many: Springer, 2016: 517-532. [20] RONNERBERGER O, FISCHER P, BROX T. U-Net: Convolutional Networks for Biomedical Image Segmentation // Proc of the International Conference on Medical Image Computing and Compu-ter-Assisted Intervention. Berlin, Germany: Springer, 2015: 234-241. [21] VEDALDI A, LENC K. MatConvNet: Convolutional Neural Networks for Matlab // Proc of the 23rd ACM International Conference on Multimedia. New York, USA: ACM, 2015: 689-692. [22] DENG J, DONG W, SOCHER R, et al. Image-Net: A Large-Scale Hierarchical Image Database // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2009: 248-255. [23] VICENTE S, CARREIRA J, AGAPITO L, et al. Reconstructing PASCAL VOC // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2014: 41-48.
2019, Vol. 32 
