Image Inpainting Based on Global-Local Prior and Texture Details
XU Qijin1, YE Hailiang1, CAO Feilong2, LIANG Jiye3
1. Department of Applied Mathematics, College of Sciences, China Jiliang University, Hangzhou 310018; 2. School of Mathematical Sciences, Zhejiang Normal University, Jinhua 321004; 3. Department of Artificial Intelligence, School of Computer and Information Technology (School of Big Data), Shanxi University, Taiyuan 030006
Abstract Image inpainting is intended to fill in missing regions of an image using surrounding information. However, existing prior-based methods often struggle to balance global semantic consistency and local texture details. In this paper, a method for image inpainting based on global-local prior and texture details is proposed. Wavelet-Fourier convolution blocks are constructed by combining wavelet convolution and Fourier convolution to enhance the interaction between local and global features. Based on the above, a global-local learning-based prior is presented. A prior extractor composed of wavelet-Fourier convolution blocks is designed to simultaneously learn global and local priors. The prior extractor is applied to both damaged and complete images to obtain damaged priors and supervised priors. During the repair phase, the damaged image and the learned priors are input into two structurally similar repair branches. Both branches are constructed with wavelet-Fourier convolutions and can simultaneously extract and fuse global and local features. Finally, the outputs of the two branches are merged to generate the image with consistent semantic content and clear local details. Additionally, a high receptive field style loss is introduced to improve image style consistency at the semantic level. Experimental results show that the proposed method outperforms existing methods on multiple datasets.
Fund:National Natural Science Foundation of China(No.62176244)
Corresponding Authors:
CAO Feilong, Ph.D., professor. His research interests include deep learning and image processing.
About author:: XU Qijin, Master student. His research interests include deep learning and image processing. YE Hailiang, Ph.D., associate professor. His research interests include deep learning and image processing. LIANG Jiye, Ph.D., professor. His research interests include artificial intelligence, granular computing, and data mining.
[1] XIANG H Y, ZOU Q, NAWAZ M A, et al. Deep Learning for Image Inpainting: A Survey. Pattern Recognition, 2023, 134. DOI: 10.1016/j.patcog.2022.109046. [2] 李月龙,高云,闫家良,等.基于深度神经网络的图像缺损修复方法综述.计算机学报, 2021, 44(11): 2295-2316. (LI Y L, GAO Y, YAN J L, et al. Image Inpainting Methods Based on Deep Neural Networks: A Review. Chinese Journal of Computers, 2021, 44(11): 2295-2316.) [3] ZHANG X B, ZHAI D H, LI T R, et al. Image Inpainting Based on Deep Learning: A Review. Information Fusion, 2023, 90: 74-94. [4] LING H, KREIS K, LI D Q, et al. EditGAN: High-Precision Semantic Image Editing // Proc of the 35th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2021: 16331-16345. [5] YANG Z P, CHU T S, LIN X, et al. Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusion. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 34(2): 1316-1320. [6] TSCHUMPERLÉ D. Fast Anisotropic Smoothing of Multi-valued Images Using Curvature-Preserving PDE's. International Journal of Compu-terVision, 2006, 68: 65-82. [7] 王相海,孙丽,万宇,等.非局域样本填充和自适应曲率驱动模型的遥感图像修复算法.模式识别与人工智能, 2016, 29(8): 735-743. (WANG X H, SUN L, WAN Y, et al. Remote Sensing Image Inpainting Based on Non-local Sample Filling and Adaptive Curvature Driven Diffusions Model. Pattern Recognition and Artificial Intelligence, 2016, 29(8): 735-743.) [8] BARNES C, ZHANG F L. A Survey of the State-of-the-Art in Patch-Based Synthesis. Computational Visual Media, 2017, 3(1): 3-20. [9] HE L T, WANG Y L. Iterative Support Detection-Based Split Bregman Method for Wavelet Frame-Based Image Inpainting. IEEE Transactions on Image Processing, 2014, 23(12): 5470-5485. [10] LIANG X, REN X, ZHANG Z D, et al. Texture Repairing by Uni-fied Low Rank Optimization. Journal of Computer Science and Technology, 2016, 31: 525-546. [11] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context Encoders: Feature Learning by Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2536-2544. [12] WAN Y C, SHAO M W, CHENG Y S, et al. Progressive Convolutional Transformer for Image Restoration. Engineering Applications of Artificial Intelligence, 2023, 125. DOI: 10.1016/j.engappai.2023.106755. [13] HUANG W L, DENG Y, HUI S Q, et al. Sparse Self-Attention Transformer for Image Inpainting. Pattern Recognition, 2024, 145. DOI: 10.1016/j.patcog.2023.109897. [14] YU X X, XU L, LI J, et al. MagConv: Mask-Guided Convolution for Image Inpainting. IEEE Transactions on Image Processing, 2023, 32: 4716-4727. [15] ZHANG R S, QUAN W Z, ZHANG Y, et al. W-Net: Structure and Texture Interaction for Image Inpainting. IEEE Transactions on Multimedia, 2023, 25: 7299-7310. [16] KIM J, KIM W, OH H, et al. Progressive Contextual Aggregation Empowered by Pixel-Wise Confidence Scoring for Image Inpain-ting. IEEE Transactions on Image Processing, 2023, 32: 1200-1214. [17] LIAO L, XIAO J, WANG Z, et al. Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 683-700. [18] QIU J J, GAO Y. Position and Channel Attention for Image Inpainting by Semantic Structure // Proc of the IEEE 32nd International Conference on Tools with Artificial Intelligence. Washington, USA: IEEE, 2020: 1290-1295. [19] LIAO H F, FUNKA-LEA G, ZHENG Y F, et al. Face Completion with Semantic Knowledge and Collaborative Adversarial Learning // Proc of the 14th Asian Conference on Computer Vision. Berlin, Germany: Springer, 2019: 382-397. [20] 邵新茹.叶海良.杨冰.等. 基于三阶段生成网络的图像修复.模式识别与人工智能, 2022, 35(12): 1047-1063. (SHAO X R, YE H L, YANG B, et al. Image Inpainting with a Three-Stage Generative Network. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1047-1063.) [21] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: Structure Gui-ded Image Inpainting Using Edge Prediction // Proc of the IEEE/CVF International Conference on Computer Vision Workshops. Washington, USA: IEEE, 2019: 3265-3274. [22] 邵杭,王永雄. 基于并行对抗与多条件融合的生成式高分辨率图像修复.模式识别与人工智能, 2020, 33(4): 363-374. (SHAO H, WANG Y X. Generative High-Resolution Image Inpainting with Parallel Adversarial Network and Multi-condition Fusion. Pattern Recognition and Artificial Intelligence, 2020, 33(4): 363-374. [23] YANG J, QI Z Q, SHI Y. Learning to Incorporate Structure Know-ledge for Image Inpainting. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12605-12612. [24] CAO C J, DONG Q L, FU Y W. ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(10): 12667-12684. [25] FINDER S E, AMOYAL R, TREISTER E, et al. Wavelet Convolutions for Large Receptive Fields // Proc of the European Confe-rence on Computer Vision. Berlin, Germany: Springer, 2024: 363-380. [26] GUO X F, YANG H Y, HUANG D. Image Inpainting via Conditional Texture and Structure Dual Generation // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 14114-14123. [27] SUVOROV R, LOGACHEVA E, MASHIKHIN A, et al. Resolution-Robust Large Mask Inpainting with Fourier Convolutions // Proc of the IEEE/CVF Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2022: 3172-3182. [28] 段立娟,武春丽,恩擎,等.基于小波域的深度残差网络图像超分辨率算法.软件学报,2019, 30(4): 941-953. (DUAN L J, WU C L, EN Q, et al. Deep Residual Network in Wavelet Domain for Image Super-Resolution. Journal of Software, 2019, 30(4): 941-953.) [29] 傅博,王洪光,宋屹峰. 融合全局和局部特征的单幅图像去雨方法.信息与控制, 2023, 52(4): 531-541. (FU B, WANG H G, SONG Y F. Single Image Deraining Method Fusing Global and Local Features. Information and Control, 2023, 52(4): 531-541.) [30] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2024-11-14]. https://arxiv.org/pdf/1409.1556v2. [31] 陶志勇,杨煜,林森. 基于语义感知的多特征协同水下图像增强.信息与控制, 2024, 53(3): 353-364. (TAO Z Y, YANG Y, LIN S. Multi-feature Collaborative Underwater Image Enhancement Based on Semantic Perception. Information and Control, 2024, 53(3): 353-364.) [32] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Ge-nerative Adversarial Nets // Proc of the 28th International Confe-rence on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2014, II: 2672-2680. [33] ZHOU B L, LAPEDRIZA A, KHOSLA A, et al. Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464. [34] KARRAS T, AILA T, LAINE S, et al. Progressive Growing of GANs for Improved Quality, Stability, and Variation[C/OL].[2024-11-14]. https://arxiv.org/pdf/1710.10196. [35] DOERSCH C, SINGH S, GUPTA A, et al. What Makes Paris Look Like Paris? ACM Transactions on Graphics, 2012, 31(4). DOI: 10.1145/2185520.2185597. [36] LIU G L, REDA F A, SHIH K J, et al. Image Inpainting for Irre-gular Holes Using Partial Convolutions // Proc of the 14th European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 85-105. [37] KINGMA D P, BA J L. ADAM: A Method for Stochastic Optimization[C/OL]. [2024-11-14]. https://arxiv.org/pdf/1412.6980. [38] LI J Y, WANG N, ZHANG L F, et al. Recurrent Feature Reaso-ning for Image Inpainting // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 7757-7765. [39] WANG D S, XIE C C, LIU S H, et al. Image Inpainting with Edge-Guided Learnable Bidirectional Attention Maps[C/OL].[2024-11-14]. https://arxiv.org/pdf/2104.12087. [40] KO K, KIM C S. Continuously Masked Transformer for Image Inpainting // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2023: 13123-13132.
2025, Vol. 38 
