Abstract In the existing sketch recognition based on deep learning, a whole sketch is employed as an input of network, and therefore the recognition process is uninterpretable. The semantic tree is introduced into sketch recognition based on deep learning, and a sketch recognition method, sketch-semantic net, is proposed in this paper. Firstly, data-driven segmentation method is utilized to divide a whole sketch into component sketches with the semantic information. Secondly, the component sketches are recognized by transfer deep learning. Finally, the component sketches are associated with the sketch categories according to the semantic concepts of the semantic tree, and thus the gap between low level semantics and high level semantics is reduced. The experimental results on the popular Sketch_ dataset demonstrate the effectiveness of the proposed method.
Fund:Supported by National Natural Science Foundation of China(No.61602004), Key Research and Development Program of Anhui Province(No.1804d8020309), Natural Science Foundation of Anhui Province(No.1908085MF188,1908085MF182), Natural Science Foundation of the Education Department of Anhui Province(No.KJ2016A041,KJ2017A011)
About author:: ZHAO Peng(Corresponding author), Ph.D., associate professor. Her research interests include intelligent information proce-ssing and machine learning.FENG Chencheng, master student. Her research interests include pattern recognition.HAN Li, Ph.D., lecturer. Her research in-terests include data mining.JI Xia, Ph.D., lecturer. Her research interests include data mining.
[1] KAZMI I K, YOU L H, ZHANG J J. A Survey of Sketch Based Modeling Systems // Proc of the 11th International Conference on Computer Graphics, Imaging and Visualization. Washington, USA: IEEE, 2014: 27-36. [2] 袁贞明,金贵朝,张 佳.基于贝叶斯网络的在线草图识别算法.计算机工程, 2010, 36(5): 32-34. (YUAN Z M, JIN G C, ZHANG J. Online Sketch Recognition Algorithm Based on Bayesian Network. Computer Engineering, 2010, 36(5): 32-34.) [3] LI B, LU Y J, SHEN J. A Semantic Tree-Based Approach for Sketch-Based 3D Model Retrieval // Proc of the 23rd International Conference on Pattern Recognition. Washington, USA: IEEE, 2017: 3880-3885. [4] EITZ M, HILDEBRAND K, BOUBEKEUR T, et al. Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(11): 1624-1636. [5] SCHNEIDER R G, TUYTELAARS T. Sketch Classification and Classification-Driven Analysis Using Fisher Vectors. ACM Transactions on Graphics, 2014, 33(6). DOI: 10.1145/2661229.2661231. [6] LI Y, HOSPEDALES T M, SONG Y Z, et al. Free-Hand Sketch Recognition by Multi-kernel Feature Learning. Computer Vision and Image Understanding, 2015, 137: 1-11. [7] SEDDATI O, DUPONT S, MAHMOUDI S. DeepSketch: Deep Con-volutional Neural Networks for Sketch Recognition and Similarity Search // Proc of the 13th International Workshop on Content-Based Multimedia Indexing. Washington, USA: IEEE, 2015. DOI: 10.1109/CBMI.2015.7153606. [8] DSEDDATI O, DUPONT S, MAHMOUDI S. DeepSketch 2: Deep Convolutional Neural Networks for Partial Sketch Recognition // Proc of the 14th International Workshop on Content-Based Multimedia Indexing. Washington, USA: IEEE, 2016. DOI: 10.1109/CBMI.2016.7500261. [9] YU Q, YANG Y X, LIU F, et al. Sketch-a-Net: A Deep Neural Network that Beats Humans. International Journal of Computer Vision, 2017, 122(3): 411-425. [10] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Cla-ssification with Deep Convolutional Neural Networks // PEREIRA F, BURGES C J C, BOTTOU L, et al., eds. Advances in Neural Information Processing Systems 25. Cambridge, USA: The MIT Press, 2012: 1097-1105. [11] SONG J F, YU Q, SONG Y Z, et al. Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 5552-5561. [12] FELLBAUM C, MILLER G A. WordNet: An Electronic Lexical Database. Cambridge, USA: The MIT Press, 1998. [13] VOORHEES E M. Using WordNet to Disambiguate Word Senses for Text Retrieval // Proc of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 1993: 171-180. [14] ASLANDOGAN Y A, THIER C, YU C T, et al. Using Semantic Contents and WordNet in Image Retrieval. ACM SIGIR Forum, 1997, 31: 286-295. [15] JIN Y H, KHAN L, WANG L, et al. Image Annotations by Combining Multiple Evidence & Wordnet // Proc of the 13th Annual ACM International Conference on Multimedia. New York, USA: ACM, 2005: 706-715. [16] TOUSCH A M, HERBIN S, AUDIBERT J Y. Semantic Hierarchies for Image Annotation: A Survey. Pattern Recognition, 2012, 45(1): 333-345. [17] PEDERSEN T, PATWARDHAN S, MICHELIZZI J. WordNet∶∶Similarity-Measuring the Relatedness of Concepts[C/OL]. [2018-11-15]. http://www.d.umn.edu/~tpederse/Pubs/AAAI04PedersenT.pdf [18] LEACOCK C, CHODOROW M. Combining Local Context and WordNet Similarity for Word Sense Identification. WordNet: An Electronic Lexical Database, 1998, 49(2): 265-283. [19] WU Z B, PALMER M. Verbs Semantics and Lexical Selection // Proc of the 32nd Annual Meeting on Association for Computational Linguistics. Stroudsburg, USA: ACL, 1994: 133-138. [20] LIN D K. An Information-Theoretic Definition of Similarity // Proc of the 15th International Conference on Machine Learning. San Francisco, USA: Morgan Kaufmann Publishers, 1998: 296-304. [21] RESNIK P. Using Information Content to Evaluate Semantic Similarity in a Taxonomy // Proc of the 14th International Joint Conference on Artificial Intelligence. New York, USA: ACM, 1995: 448-453. [22] JIANG J J, CONRATH D W. Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy // Proc of the International Confe-rence on Research in Computational Linguistics. Berlin, Germany: Springer, 1997: 19-33. [23] HIRST G, ST-ONGE D. Lexical Chains as Representations of Context for the Detection and Correction of Malapropisms. Lecture Notes in Physics, 1997, 728(9): 123-149. [24] BANERJEE S, PEDERSEN T. Extended Gloss Overlaps as a Mea-sure of Semantic Relatedness[C/OL]. [2018-11-15]. http://www.d.umn.edu/~tpederse/Pubs/ijcai03.pdf. [25] PEDERSEN T, BANERJEE S, PATWARDHAN S. Maximizing Semantic Relatedness to Perform Word Sense Disambiguation. Research Report, UMSI 2005/25. Duluth, USA: University of Minnesota, 2005. [26] HUANG Z, FU H B, LAU R W H. Data-Driven Segmentation and Labeling of Freehand Sketches. ACM Transactions on Graphics, 2014, 33(6). DOI: 10.1145/2661229.2661280. [27] 赵 鹏,王 斐,刘慧婷,等.基于深度学习的手绘草图识别.四川大学学报(工程科学版), 2016, 48(3): 94-99. (ZHAO P, WANG F, LIU H T, et al. Sketch Recognition Using Deep Learning. Journal of Sichuan University(Engineering Science Edition), 2016, 48(3): 94-99.) [28] 赵 鹏,刘 杨,刘慧婷,等.基于深度卷积-递归神经网络的手绘草图识别方法.计算机辅助设计与图形学学报, 2018, 30(2): 217-224. (ZHAO P, LIU Y, LIU H T, et al. A Sketch Recognition Method Based on Deep Convolutional-Recurrent Neural Network. Journal of Computer-Aided Design and Computer Graphics, 2018, 30(2): 217-224.)
2019, Vol. 32 
