Abstract As the missing information in the image is increasing, the existing methods extracting information from only a single image can not produce satisfactory completion results. Therefore, an automatic label conditional generative adversarial network(CGAN) based on image semantic is presented from the perspective of multi-granular cognition. It can be applied on image denoising and image completion. Firstly, the multi-layer semantic information from unlabeled images based on the Gaussian cloud transform algorithm is extracted. Then, the original images are segmented and the segmented images are labeled automatically in accordance with different granular semantic information. Furthermore, different granular segmented images and their labels are used as the training samples in the CGAN to get an image probability generation model, respectively. The large missing regions from a single image are completed based on the similar image generated by cloud semantic and CGAN. On the datasets of Caltech-UCSD Birds and Oxford-102flowers, the proposed model achieves the high performance in image denoising and image completion.
Fund:Supported by National Natural Science Foundation of China(No.61572091)
Corresponding Authors:
DU Qiuping(Corresponding author), master student. His research interests include could model, image processing and deep learning.
About author:: LIU Qun, Ph.D., professor. Her research interests include nonlinear dynamics, artificial neural network and complex networks.
[1] DOSOVITSKIY A, SPRINGENBERG J T, BROX T. Learning to Generate Chairs with Convolutional Neural Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 1538-1546. [2] REED S E, ZHANG Y, ZHANG Y T, et al. Deep Visual Analogy-Making // CORTES C, LASORENCE N O, LEE D D, et al., eds. Advances in Neural Information Processing Systems 28. Cambridge, USA: The MIT Press, 2015: 1252-1260. [3] VAN DEN OORD A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel Recurrent Neural Networks // Proc of the International Confe-rence on Machine Learning. New York, USA: ACM, 2016: 1747-1756. [4] GREGOR K, DANIHELKA I, GRAVES A, et al. DRAW: A Recurrent Neural Network for Image Generation // Proc of the 32nd International Conference on Machine Learning. New York, USA: ACM, 2015: 1462-1471. [5] KINGMA D P, WELLING M. Auto-Encoding Variational Bayes[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1312.6114.pdf. [6] YAN X C, YANG J M, SOHN K, et al. Attribute2Image: Conditional Image Generation from Visual Attributes // Proc of the European Conference on Computer Vision. Amsterdam, The Netherlands: Springer International Publishing, 2016: 776-791. [7] GOODFELLOW I J, POUGETABADIE J, MIRZA M, et al. Generative Adversarial Nets[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1406.2661v1.pdf. [8] METZ L, POOLE B, PFAU D, et al. Unrolled Generative Adversarial Networks[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1611.02163v1.pdf. [9] HOCHREITER S, SCHMIDHUBER J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780. [10] ZHAO F, FENG J S, ZHAO J, et al. Robust LSTM-Autoencoders for Face De-Occlusion in the Wild. IEEE Transaction on Image Processing, 2018, 27(2): 778-790. [11] LEDIG C, THEIS L, HUSZÁR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1609.04802.pdf. [12] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1611.07004.pdf. [13] MIRZA M, OSINDERO S. Conditional Generative Adversarial Nets[C/OL].[2017-08-20]. https://arkiv.org/pdf/1411.1784.pdf. [14] REED S, AKATA Z, YAN X C, et al. Generative Adversarial Text to Image Synthesis // Proc of the 33rd International Confe-rence on Machine Learning. New York, USA: ACM, 2016: 1060-1069. [15] ZHANG H, XU T, LI H S, et al. StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1612.03242v1.pdf. [16] AFONSO M V, BIOUCAS-DIAS J M, FIGUEIREDO M A T. An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems. IEEE Transactions on Image Processing, 2011, 20(3): 681-695. [17] SHEN J H, CHAN T F. Mathematical Models for Local Nontexture Inpaintings. SIAM Journal on Applied Mathematics, 2001, 62(3): 1019-1043. [18] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Transactions on Graphics, 2009, 28(3): 1-11. [19] PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context Encoders: Feature Learning by Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016:2536-2544. [20] 李德毅,孟海军,史雷梅.隶属云和隶属云发生器.计算机研究与发展, 1995, 32(6): 15-20. (LI D Y, MENG H J, SHI L M. Membership Clouds and Membership Cloud Generators. Journal of Computer Research and Development, 1995, 32(6): 15-20.) [21] 李德毅,杜 鹢.不确定性人工智能.第2版.北京:国防工业出版社, 2014. (LI D Y, DU Y. Artificial Intelligence with Uncertainty. 2nd Edi- tion. Beijing, China: National Defense Industry Press, 2014.) [22] 王坤峰,苟 超,段艳杰,等.生成式对抗网络GAN的研究进展与展望.自动化学报, 2017, 43(3): 321-332. (WANG K F, GOU C, DUAN Y J, et al. Generative Adversarial Networks: The State of the Art and Beyond. Acta Automatica Sinica, 2017, 43(3): 321-332.) [23] 马鸿耀,王国胤,张清华,等.基于云模型的多粒度彩色图像分割.计算机工程, 2012, 38(20): 184-187. (MA H Y, WANG G Y, ZHANG Q H, et al. Multi-granularity Color Image Segmentation Based on Cloud Model. Computer Engineering, 2012, 38(20): 184-187.) [24] 姚 红,王国胤,张清华.基于粗糙集和云模型的彩色图像分割方法.小型微型计算机系统, 2013, 34(11): 2615-2620. (YAO H, WANG G Y, ZHANG Q H. Color Image Segmentation Method Based on Rough Sets and Cloud Model. Journal of Chinese Computer System, 2013, 34(11): 2615-2620.) [25] GOODFELLOW I. NIPS 2016 Tutorial: Generative Adversarial Networks[C/OL]. [2017-08-20]. https://arxiv.org/pdf/1701.00160.pdf. [26] WAH C, BRANSON S, WELINDER P, et al. The Caltech-UCSD Birds-200-2011 Dataset[C/OL]. [2017-08-20]. https://authors.library.caltech.edu/27452/1/CUB_200_2011.pdf. [27] NILSBACK M E, ZISSERMAN A. Automated Flower Classification over a Large Number of Classes // Proc of the 6th Indian Confe-rence on Computer Vision, Graphics & Image Processing. Washington, USA: IEEE, 2008: 722-729.
2018, Vol. 31 
