Edge-Texture Dual Feature Aggregation for Image Inpainting via Structural Transformation Completion
ZHANG Rongguo1, WEN Yihao1, HU Jing1, WANG Lifang1, LIU Xiaojun2
1. College of Computer Science and Technology, Taiyuan University of Science and Technology, Taiyuan 030024;
2. School of Mechanical Engineering, Hefei University of Technology, Hefei 230009
The deficiencies in restoring plausible edge structures and complete textures within missing regions still emerge in existing neural network-based approaches for image inpainting. To address these issues, a method for edge-texture dual feature aggregation for image inpainting via structural transformation completion(ETSTC) is proposed. First, a structure transform completer module integrating axial attention and contextual transformer is designed. The module is combined with a structure smoother module to further complement and optimize edge structures. Thus, both local edge details and global structural patterns are effectively captured while edge noise and artifacts are suppressed. Second, an edge-guided feature aligner and an edge-texture dual-feature aggregator are developed. Scaling and offset parameters are adaptively learned to effectively resolve scale and offset discrepancies in dynamic aggregation of edge structural features and texture features across different feature space levels, and thereby the image inpainting performance is improved. Finally, experiments on three datasets verify the feasibility and effectiveness of ETSTC.
[1] YANG J, RUHAIYEM N I R. Review of Deep Learning-Based Image Inpainting Techniques. IEEE Access, 2024, 12: 138441-138482.
[2] WALI A, NASEER A, TAMOOR M, et al. Recent Progress in Di-gital Image Restoration Techniques: A Review. Digital Signal Processing, 2023, 141. DOI: 10.1016/j.dsp.2023.104187.
[3] XIANG H Y, ZOU Q, NAWAZ M A, et al. Deep Learning for Image Inpainting: A Survey. Pattern Recognition, 2023, 134. DOI: 10.1016/j.patcog.2022.109046.
[4] CHEN Y T, XIA R L, YANG K, et al. DNNAM: Image Inpainting Algorithm via Deep Neural Networks and Attention Mechanism. Applied Soft Computing, 2024, 154. DOI: 10.1016/j.asoc.2024.111392.
[5] YU X X, XU L, LI J, et al. MagConv: Mask-Guided Convolution for Image Inpainting. IEEE Transactions on Image Processing, 2023, 32: 4716-4727.
[6] WAN Z Y, ZHANG J B, CHEN D D, et al. High-Fidelity Pluralistic Image Completion with Transformers // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 4672-4681.
[7] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: Structure Guided Image Inpainting Using Edge Prediction // Proc of the IEEE/CVF International Conference on Computer Vision Workshops. Washington, USA: IEEE, 2019: 3265-3274.
[8] TYLEI?EK R, ŠÁRA R. Spatial Pattern Templates for Recognition of Objects with Regular Structure // Proc of the 35th German Conference on Pattern Recognition. Berlin, Germany: Springer, 2013: 364-374.
[9] LIU Z W, LUO P, WANG X G, et al. Deep Learning Face Attri-butes in the Wild // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 3730-3738.
[10] ZHOU B L, LAPEDRIZA A, KHOSLA A, et al. Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464.
[11] SHEN J H, KANG S H, CHAN T F.Euler's Elastica and Curvature-Based Inpainting. SIAM Journal on Applied Mathematics, 2003, 63(2): 564-592.
[12] TSCHUMPERLÉ D, DERICHE R.Vector-Valued Image Regularization with PDEs: A Common Framework for Different Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(4): 506-517.
[13] CRIMINISI A, PÉREZ P, TOYAMA K. Region Filling and Object Removal by Exemplar-Based Image Inpainting. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.
[14] RU?I? T, PI?URICA A. Context-Aware Patch-Based Image Inpainting Using Markov Random Field Modeling. IEEE Transactions on Image Processing, 2015, 24(1): 444-456.
[15] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context Encoders: Feature Learning by Inpainting // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2536-2544.
[16] CHEN G, ZHANG G P, YANG Z G, et al. Multi-scale Patch-GAN with Edge Detection for Image Inpainting. Applied Intelligence, 2023, 53(4): 3917-3932.
[17] WANG Y F, GUO D S, ZHAO H R, et al. Image Inpainting via Multi-scale Adaptive Priors. Pattern Recognition, 2025, 162. DOI: 10.1016/j.patcog.2025.111410.
[18] 邵新茹.叶海良.杨冰.等. 基于三阶段生成网络的图像修复.模式识别与人工智能, 2022, 35(12): 1047-1063.
(SHAO X R, YE H L, YANG B, et al. Image Inpainting with a Three-Stage Generative Network. Pattern Recognition and Artificial Intelligence, 2022, 35(12): 1047-1063.)
[19] GUO X F, YANG H Y, HUANG D.Image Inpainting via Conditional Texture and Structure Dual Generation // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 14114-14123.
[20] LIU Q K, JIANG Y Q, TAN Z T, et al. Transformer Based Plura-listic Image Completion with Reduced Information Loss. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(10): 6652-6668.
[21] WANG Q, WANG Z C, ZHANG X P, et al. Art Image Inpainting with Style-Guided Dual-Branch Inpainting Network. IEEE Transactions on Multimedia, 2024, 26: 8026-8037.
[22] LI W B, LIN Z, ZHOU K, et al. MAT: Mask-Aware Transformer for Large Hole Image Inpainting // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 10748-10758.
[23] KO K, KIM C S.Continuously Masked Transformer for Image Inpainting // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2023: 13123-13132.
[24] ZHANG C, YANG W X, LI X, et al. MMGInpainting: Multi-modality Guided Image Inpainting Based on Diffusion Models. IEEE Transactions on Multimedia, 2024, 26: 8811-8823.
[25] HUAN L X, XUE N, ZHENG X W, et al. Unmixing Convolutional Features for Crisp Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6602-6609.
[26] CAO C J, FU Y W.Learning a Sketch Tensor Space for Image Inpainting of Man-Made Scenes // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 14489-14498.
[27] WANG H Y, ZHU Y K, GREEN B, et al. Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 108-126.
[28] LI Y H, YAO T, PAN Y W, et al. Contextual Transformer Networks for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(2): 1489-1500.
[29] KOLODOCHKA D, POLYAKOVA S M, NESTERIUK P D O, et al. LaMa Network Architecture Search for Image Inpainting[C/OL].[2025-03-17]. https://ceur-ws.org/Vol-3790/paper32.pdf.
[30] HU J, SHEN L, SUN G.Squeeze-and-Excitation Networks // Proc of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 7132-7141.
[31] PARK T, LIU M Y, WANG T C, et al. Semantic Image Synthesis with Spatially-Adaptive Normalization // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 2332-2341.
2025, Vol. 38 
