|
|
|
| Generative High-Resolution Image Inpainting with Parallel Adversarial Network and Multi-condition Fusion |
| SHAO Hang1, WANG Yongxiong1 |
| 1.Department of Automation, School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai 200093 |
|
|
|
|
Abstract Regions with artifacts and semantic inaccuracy are often caused by existing image inpainting algorithms. Moreover, the inpainting effect is limited for images with large missing regions and high-resolution. Therefore, a two-stage image inpainting approach based on parallel adversarial network and multi-condition fusion is proposed in this paper. Firstly, an improved deep residual network is utilized to fill the corrupted image. The first-stage adversarial network is employed to complete the image edge map. Next, the color feature of the filled image is extracted and fused with the completed edge image. Then, the fused image is applied as the condition label of the second-stage adversarial network. Finally, the inpainting result is obtained by the second-stage network with a contextual attention module. Experiments on multiple public datasets demonstrate that realistic inpainting results can be obtained by the proposed approach.
|
|
Received: 10 January 2020
|
|
|
| Fund:Supported by National Natural Science Foundation of China(No. 61673276) |
|
Corresponding Authors:
SHAO Hang, master student. His research interests include pattern recognition, image processing and computer vision.
|
| About author:: SHAO Hang, master student. His research interests include pattern recognition, image processing and computer vision. |
|
|
|
[1] KOMODAKIS N, TZIRITAS G. Image Completion Using Efficient Belief Propagation via Priority Scheduling and Dynamic Pruning. IEEE Transactions on Image Processing, 2007, 16(11): 2649-2661.
[2] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. ACM Transactions on Graphics, 2009, 28(3): 24-33.
[3] 李旭峰,王 静,刘红敏,等.特征优先块匹配图像修复算法.计算机辅助设计与图形学学报, 2016, 28(7): 1131-1137.
(LI X F, WANG J, LIU H M, et al. Image Inpainting Using Feature Precedence and Patch Matching. Journal of Computer-Aided Design and Computer Graphics, 2016, 28(7): 1131-1137.)
[4] HAYS J, EFROS A A. Scene Completion Using Millions of Photographs. ACM Transactions on Graphics, 2007, 26(3): 4-10.
[5] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context Encoders: Feature Learning by Inpainting // Proc of the IEEE Confe-
rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2536-2544.
[6] YANG C, LU X, LIN Z, et al. High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 6721-6729.
[7] 林懿伦,戴星原,李 力,等.人工智能研究的新前线:生成式对抗网络.自动化学报, 2018, 44(5): 775-792.
(LIN Y L, DAI X Y, LI L, et al. The New Frontier of AI Research: Generative Adversarial Networks. Acta Automatica Sinica, 2018, 44(5): 775-792.)
[8] YU J H, LIN Z, YANG J M, et al. Generative Image Inpainting with Contextual Attention // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 5505-5514.
[9] ZENG Y H, FU J L, CHAO H Y, et al. Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 1486-1494.
[10] LI J Y, HE F X, ZHANG L F, et al. Progressive Reconstruction of Visual Structure for Image Inpainting // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2019: 5961-5970.
[11] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and Locally Consistent Image Completion. ACM Transactions on Graphics, 2017, 36(4): 107-120.
[12] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning[C/OL]. [2019-12-26]. https://arxiv.org/pdf/1901.00212.pdf.
[13] XIONG W, YU J, LIN Z M, et al. Foreground-Aware Image Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 5840-5848.
[14] SAGONG M C, SHIN Y G, KIM S W, et al. PEPSI: Fast Image Inpainting with Parallel Decoding Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 11360-11368.
[15] REN Y R, YU X M, ZHANG R N, et al. StructureFlow: Image Inpainting via Structure-Aware Appearance Flow // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2019: 181-190.
[16] XU L, YAN Q, XIA Y, et al. Structure Extraction from Texture via Relative Total Variation. ACM Transactions on Graphics, 2012, 31(6): 139-148.
[17] ZHOU B L, LAPEDRIZA A, KHOSLA A, et al. Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(6): 1452-1464.
[18] LIU Z W, LUO P, WANG X G, et al. Deep Learning Face Attributes in the Wild // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 3730-3738.
[19] ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-Image Translation with Conditional Adversarial Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1125-1134.
[20] PHILBIN J, CHUM O, ISARD M, et al. Object Retrieval with Large Vocabularies and Fast Spatial Matching // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2007. DOI: 10.1109/CVPR.2007.383172.
[21] BALLESTER C, BERTALMIO M, CASELLES V, et al. Filling-in by Joint Interpolation of Vector Fields and Gray Levels. IEEE Transactions on Image Processing, 2001, 10(8): 1200-1211.
[22] LEVIN A, ZOMET A, WEISS Y. Learning How to Inpaint from Global Image Statistics // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2003. DOI: 10.1109/ICCV.2003.1238360.
[23] DRORI I, COHEN-OR D, YESHURUN H. Fragment-Based Image Completion. ACM Transaction on Graphics, 2003, 22(3): 303-312.
[24] KWATRA V, ESSA I, BOBICK A, et al. Texture Optimization for Example-Based Synthesis. ACM Transaction on Graphics, 2005, 24(3): 795-802.
[25] SIMAKOV D, CASPI Y, SHECHTMAN E, et al. Summarizing Visual Data Using Bidirectional Similarity // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2008. DOI: 10.1109/CVPR.2008.4587842.
[26] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Classification with Deep Convolutional Neural Networks // PEREIRA F, BURGES C J C, BOTTOU L ,et al., eds. Advances in Neural Information Processing Systems 25. Cambridge, USA: The MIT Press, 2012: 1097-1105.
[27] LI Y J, LIU S F, YANG J M, et al. Generative Face Completion // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 3911-3919.
[28] YU J H, LIN Z, YANG J M, et al. Free-Form Image Inpainting with Gated Convolution // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2019: 4471-4480.
[29] XIE C H, LIU S H, LI C, et al. Image Inpainting with Learnable Bidirectional Attention Maps // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2019: 8858-8867.
[30] LIU H Y, JIANG B, XIAO Y, et al. Coherent Semantic Attention for Image Inpainting // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2019: 4170-4179.
[31] 杜秋平,刘 群.基于图像云模型语义标注的条件生成对抗网络.模式识别与人工智能, 2018, 31(4): 379-388.
(DU Q P, LIU Q. Conditional Generative Adversarial Network Based on Image Semantic Annotation of Cloud Model. Pattern Recognition and Artificial Intelligence, 2018, 31(4): 379-388.)
[32] ZHANG R, ISOLA P, EFROS A A. Colorful Image Colorization // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 649-666.
[33] 张焯林,赵建伟,曹飞龙.构建带空洞卷积的深度神经网络重建高分辨率图像.模式识别与人工智能, 2019, 32(3): 259-267.
(ZHANG Z L, ZHAO J W, CAO F L. Building Deep Neural Networks with Dilated Convolutions to Reconstruct High-Resolution Image. Pattern Recognition and Artificial Intelligence, 2019, 32(3): 259-267.)
[34] LEDIG C, THEIS L, HUSZAR F, et al. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 4681-4690.
[35] JOHNSON J, ALASI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 694-711.
[36] SAJJADI M S M, SCHÖLKOPF B, HIRSCH M. EnhanceNet: Single Image Super-Resolution through Automated Texture Synthesis // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4491-4500.
[37] LIU G L, REDA F A, SHIH K J, et al. Image Inpainting for Irregular Holes Using Partial Convolutions // Proc of the European Con-
ference on Computer Vision. Berlin, Germany: Springer, 2018: 85-100. |
|
|
|
2020, Vol. 33 
