Saliency Background Guided Network for Weakly-Supervised Semantic Segmentation
BAI Xuefei1, LI Wenjing1, WANG Wenjian1,2
1. School of Computer and Information Technology, Shanxi University, Taiyuan 030006 2. Key Laboratory Computational Intelligence and Chinese Information Processing of Ministry of Education, Shanxi University, Taiyuan 030006
Abstract Weakly-supervised semantic segmentation methods based on image-level annotation mostly rely on the initial response of class activation map to locate the segmented object region. However, the class activation map only focuses on the most discriminative area of the object, and the shortcomings exit, including small target area and blurred boundary. Therefore, the final segmentation result is incomplete. To overcome this problem, a saliency background guided network for weakly-supervised semantic segmentation is proposed. Firstly, the background seed region is generated through image saliency mapping and background iteration, and then it is fused with the class activation map generated by the classification network. Thus, effective pseudo pixel labels for training the semantic segmentation model are obtained. The segmentation process does not entirely depend on the most discriminative object. The information complementation is implemented through image saliency background features and class activation response map. Consequently, pixel labels are more accurate, and the performance of the segmentationnetwork is improved.Experiments on PASCALVOC 2012 dataset verify the effectiveness of the proposed method. Moreover, the proposed method makes a significant improvement in segmentation performance.
Fund:National Natural Science Foundation of China(No.61703252,62076154), Key Research and Development Program of International Cooperation of Shanxi Province(No.201903D421 050), Local Science and Technology Innovation Project Guided by Central Government(No.YDZX20201400001224)
Corresponding Authors:
WANG Wenjian, Ph.D., professor. Her research interests include machine learning, computing intelligence and image processing.
About author:: BAI Xuefei, Ph.D., associate professor. Her research interests include image proce-ssing and machine learning. LI Wenjing, master student. Her research interests include image processing and machine learning.
[1] WANG X, MA H M, YOU S Y. Deep Clustering for Weakly-Supervised Semantic Segmentation in Autonomous Driving Scenes. Neurocomputing, 2020, 381: 20-28. [2] IMTIAZ R, KHAN T M, NAQVI S S, et al. Screening of Glaucoma Disease From Retinal Vessel Images Using Semantic Segmentation. Computers and Electrical Engineering, 2021, 91. DOI: 10.1016/j.compeleceng.2021.107036. [3] ZHOU L, KONG X Y, GONG C, et al. FC-RCCN: Fully Convolutional Residual Continuous CRF Network for Semantic Segmentation. Pattern Recognition Letters, 2020, 130: 54-63. [4] TAO H, LI W H, QIN X X, et al. Image Semantic Segmentation Based on Convolutional Neural Network and Conditional Random Field // Proc of the 10th International Conference on Advanced Computational Intelligence. Washington, USA: IEEE, 2018: 568-572. [5] WU H S, LIANG C X, LIU M S, et al. Optimized HRNet for Image Semantic Segmentation. Expert Systems with Applications, 2021, 174. DOI: 10.1016/j.eswa.2020.114532. [6] KHOREVA A, BENENSON R, HOSANG J, et al. Simple Does It: Weakly Supervised Instance and Semantic Segmentation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1665-1674. [7] PAPANDREOU G, CHEN L C, MURPHY K P, et al. Weakly-and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 1742-1750. [8] BEARMAN A, RUSSAKOVSKY O, FERRARI V, et al. What′s the Point: Semantic Segmentation with Point Supervision // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 549-565. [9] ALEXE B, DESELAERS T, FERRARI V. Measuring the Object-ness of Image Windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(11): 2189-2202. [10] PU M Y, HUANG Y P, GUAN Q J, et al. GraphNet: Learning Image Pseudo Annotations for Weakly-Supervised Semantic Segmentation // Proc of the 26th ACM International Conference on Multimedia. Washington, USA: ACM, 2018: 483-491. [11] VERNAZA P, CHANDRAKER M. Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2953-2961. [12] ZHU Y, ZHOU Y Z, XU H J, et al. Learning Instance Activation Maps for Weakly Supervised Instance Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019, I: 3111-3120. [13] WAN F, LIU C, KE W, et al. C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 2194-2203. [14] PINHEIRO P O, COLLOBERT R. From Image-Level to Pixel-Level Labeling with Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 1713-1721. [15] EVERINGHAM M, ESLAMI S M A, VAN GOOL L, et al. The PASCAL Visual Object Classes Challenge: A Retrospective. International Journal of Computer Vision, 2015, 111(1): 95-136. [16] ZHOU B L, KHOSL A, LAPEDRIZA A, et al. Learning Deep Features for Discriminative Localization // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2921-2929. [17] KOLESNIKOV A, LAMPERT C H. Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 695-711. [18] HUANG Z L, WANG X G, WANG J S, et al. Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 7014-7023. [19] SUN G L, WANG W G, DAI J F, et al. Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 347-365. [20] PAN S Y, LU C Y, LEE S P, et al. Weakly-Supervised Image Semantic Segmentation Using Graph Convolutional Networks // Proc of the IEEE International Conference on Multimedia and Expo. Washington, USA: IEEE, 2021. DOI: 10.1109/ICME51207.2021.9428116. [21] SHIMODA W, YANAI K. Distinct Class-Specific Saliency Maps for Weakly Supervised Semantic Segmentation // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 218-234. [22] AHN J, KWAK S. Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 4981-4990. [23] WANG X, YOU S D, LI X, et al. Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 1354-1362. [24] OH S J, BENENSON R, KHOREVA A, et al. Exploiting Saliency for Object Segmentation from Image Level Labels // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 5038-5047. [25] SUN F D, LI W H. Saliency Guided Deep Network for Weakly-Supervised Image Segmentation. Pattern Recognition Letters, 2019, 120: 62-68. [26] 李 阳,刘 扬,刘国军,等.基于对象位置线索的弱监督图像语义分割方法.软件学报, 2020, 31(11): 3640-3656. (LI Y, LIU Y, LIU G J, et al. Weakly Supervised Image Semantic Segmentation Method Based on Object Location Cues. Journal of Software, 2020, 31(11): 3640-3656.) [27] LEE D H. Pseudo-Label: The Simple and Efficient Semi-supervised Learning Method for Deep Neural Networks. [C/OL]. [2021-04-27] . http://deeplearning.net/wp-content/uploads/2013/03/pseudo_label_final.pdf. [28] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848. [29] GUAN Q, WANG Y J, PING B, et al. Deep Convolutional Neural Network VGG-16 Model for Differential Diagnosing of Papillary Thyroid Carcinomas in Cytological Images: A Pilot Study. Journal of Cancer, 2019, 10(20): 4876-4882. [30] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [31] LIN D, DAI J F, JIA J Y, et al. ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 3159-3167. [32] JIANG P T, HOU Q B, CHENG M M, et al. Integral Object Mining via Online Attention Accumulation // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2019: 2070-2079. [33] GAO S H, CHENG M M, ZHAO K, et al. Res2Net: A New Multi-scale Backbone Architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(2): 652-662. [34] SHELHAMER E, LONG J, DARRELL T. Fully Convolutional Net-works for Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 39(4): 640-651. [35] CHENG M M, MITRA N J, HUANG X L, et al. Global Contrast Based Salient Region Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 569-582.
2021, Vol. 34 
