Chinese Paintings Sentiment Recognition via CNN Optimization with Human Cognition
SHENG Jiachuan1, CHEN Yaqi1, WANG Jun2, LI Liang3,4
1. School of Science and Technology, Tianjin University of Finance and Economics, Tianjin 300222; 2. Department of Management Information Systems, Tianjin University of Finance and Economics, Tianjin 300222; 3. Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering, China Three Gorges University, Yichang 443002; 4. College of Intelligence and Computing, Tianjin University, Tianjin 300350
Abstract Most existing methods are lack of analysis of Chinese paintings sentiment. In this paper, an algorithm of Chinese paintings sentiment recognition via convolutional neural network (CNN) optimization with human cognition is proposed. Firstly, the region of interest from painting is extracted to obtain the square area with rich emotional expression according to image saliency and brushstroke complexity. Then, the deep learning network structure is optimized by combining feature visualization and the knowledge of emotional expression techniques in traditional Chinese paintings. Finally, a rebuilt network is fine-tuned for Chinese paintings sentiment classification task. In the experiment, 1000 Chinese paintings are classified with four emotions and the accuracy rate is better than that of other convolutional neural networks. The effect of model operation is explained through ablation and visualization experiments, and thus the ability of the proposed algorithm to recognize the sentiment of traditional Chinese paintings is confirmed.
Fund:Supported by National Natural Science Foundation of China(No.61502331,61602338), Foundation for Innovative Research Groups of Hubei Province(No.2015CFA025), MOE Project of Humanities and Social Sciences(No.18YJA63005T), Natural Science Foundation of Tianjin(No.18JCYBJC85100)
Corresponding Authors:
LI Liang, Ph.D., lecturer. His research interests include digital multimedia processing and pattern recognition.
About author:: SHENG Jiachuan, Ph.D., associate professor. Her research interests include digital multimedia processing and pattern recognition; WANG Jun, Ph.D., associate professor. His research interests include intelligent optimization algorithm and application; CHEN Yaqi, master student. Her research interests include digital multimedia processing and machine learning.
[1] HE X Y, ZHANG W. Emotion Recognition by Assisted Learning with Convolutional Neural Networks. Neurocomputing, 2018, 291: 187-194. [2] KANG D W, SHIM H, YOON K. A Method for Extracting Emotion Using Colors Comprise the Painting Image. Multimedia Tools and Applications, 2018, 77(4): 4985-5002. [3] 贾春花,郭小英,白茹意.绘画特征提取方法与情感分析研究综述.中国图象图形学报, 2018, 23(7): 937-952. (JIA C H, GUO X Y, BAI R Y. Review of Feature Extraction Methods and Research on Affective Analysis for Paintings. Journal of Image and Graphics, 2018, 23(7): 937-952.) [4] 刘 赏,盛家川.基于线条形状和主方向墨色分布特征的中国画画家识别算法.模式识别与人工智能, 2017, 30(10): 917-927. (LIU S, SHENG J C. Artists Recognition via Line Shape and Ink Color Distribution of the Principal Direction for Chinese Paintings. Pattern Recognition and Artificial Intelligence, 2017, 30(10): 917-927.) [5] 盛家川,李玉芝.国画的艺术目标分割及深度学习与分类.中国图象图形学报, 2018, 23(8): 1193-1206. (SHENG J C, LI Y Z. Learning Artistic Objects for Improved Cla-ssification of Chinese Paintings. Journal of Image and Graphics, 2018, 23(8): 1193-1206.) [6] 李玉芝,盛家川,华 斌.中国画分类的改进嵌入式学习算法.计算机辅助设计与图形学学报, 2018, 30(5): 893-900. (LI Y Z, SHENG J C, HUA B. Improved Embedded Learning for Classification of Chinese Paintings. Journal of Computer-Aided Design and Computer Graphics, 2018, 30(5): 893-900.) [7] 王 征,李皓月,许洪山,等.基于卷积神经网络和SVM的中国画情感分类.南京师大学报(自然科学版), 2017, 40(3): 74-79, 86. (WANG Z, LI H Y, XU H S, et al. Chinese Painting Emotion Classification Based on Convolution Neural Network and SVM. Journal of Nanjing Normal University(Natural Science Edition), 2017, 40(3): 74-79, 86.) [8] 樊 迪,HYUNWOO K,陈晓鹏,等.机器仿生眼的多任务学习人脸分析.模式识别与人工智能, 2019, 32(1): 10-16. (FAN D, HYUNWOO K, CHEN X P, et al. Multi-task Learning Based Face Analysis for Machine Bionic Eyes. Pattern Recognition and Artificial Intelligence, 2019, 32(1): 10-16.) [9] 何姣姣,张永平,姚拓中,等.基于并行通道级联网络的鲁棒行人检测.模式识别与人工智能, 2018, 31(12): 1134-1142. (HE J J, ZHANG Y P, YAO T Z, et al. Robust Pedestrian Detection Based on Parallel Channel Cascade Network. Pattern Recognition and Artificial Intelligence, 2018, 31(12): 1134-1142.) [10] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2019-02-26]. https://arxiv.org/pdf/1409.1556.pdf. [11] SZEGEDY C, LIU W, JIA Y Q, et al. Going Deeper with Convolutions // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015. DOI: 10.1109/CVPR.2015.7298594. [12] 李永忠.八大山人与石涛绘画题款的比较.中国书法, 2018(17): 88-99. (LI Y Z. Comparison between Badashanren and Shi Tao's Painting Inscription. Chinese Calligraphy, 2018(17): 88-99.) [13] ZHOU B L, KHOSLA A, LAPEDRIZA A, et al. Learning Deep Features for Discriminative Localization // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2921-2929. [14] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Loca-lization // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 618-626. [15] OLAH C, MORDVINTSEV A, SCHUBERT L. Feature Visualization[J/OL]. [2019-02-26]. https://distill.pub/2017/feature-visualization. [16] OLAH C, SATYANARAYAN A, JOHNSON I, et al. The Building Blocks of Interpretability[J/OL]. [2019-02-26]. https://distill.pub/2018/building-blocks. [17] FU R G, LI B, GAO Y H, et al. Visualizing and Analyzing Convolution Neural Networks with Gradient Information. Neurocomputing, 2018, 293: 12-17. [18] CETINIC E, LIPIC T, GRGIC S, et al. Fine-Tuning Convolutional Neural Networks for Fine Art Classification. Expert Systems with Applications, 2018, 114: 107-118. [19] FALOMIR Z, MUSEROS L, SANZ I, et al. Categorizing Paintings in Art Styles Based on Qualitative Color Descriptors, Quantitative Global Features and Machine Learning(QArt-Learn). Expert Systems with Applications, 2018, 97: 83-94. [20] 郭 艺.中国画画幅形式的变革与创新.美术观察, 2015(2): 128-129. (GUO Y. The Transformation and Innovation of Chinese Painting Format. Art Observation, 2015(2): 128-129.) [21] ZHANG J M, SCLAROFF S. Saliency Detection: A Boolean Map Approach // Proc of the IEEE International Conference on Compu-ter Vision. Washington, USA: IEEE, 2013: 153-160. [22] ZHANG J M, SCLAROFF S. Exploiting Surroundedness for Saliency Detection: A Boolean Map Approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(5): 889-902. [23] ERHAN D, BENGIO Y, COURVILLE A, et al. Visualizing Higher-Layer Features of a Deep Network. Technical Report, 1341. Montreal, Canada: University of Montreal, 2009.
2020, Vol. 33 
