基于差异双分支编码器的多阶段图像融合方法

doi:10.16451/j.cnki.issn1003-6059.202207008

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (2952 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要在现有的红外和可见光图像融合方法中,融合图像中的细节信息丢失严重,视觉效果不佳.针对上述问题,文中提出基于差异双分支编码器的多阶段图像融合方法.通过两支不同结构的编码器提取多模态图像的特征,增强特征的多样性.设计多阶段的融合策略,实现精细化图像融合.首先,在差异双分支编码器中,对两个编码分支提取的差异性特征进行初级融合.然后,在融合阶段,对多模态图像的显著性特征进行中级融合.最后,使用远程横向连接将差异双分支编码器的浅层特征传送给解码器,同时指导融合过程和图像重建.对比实验表明,文中算法可增强融合图像的细节信息,并在视觉效果和客观评价上都较优.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	洪雨露
	吴小俊
	徐天阳

关键词 ：图像融合, 红外图像, 可见光图像, 卷积神经网络

Abstract：In the existing infrared and visible image fusion methods, the details of the fused image are lost seriously and the visual effect is poor. Aiming at the problems, a multi-stage image fusion method based on differential dual-branch encoder is proposed. The features of multi-modal images are extracted by two encoders with different network structures to enhance the diversity of features. A multi-stage fusion strategy is designed to achieve refined image fusion. Firstly, primary fusion is performed on the differential features extracted by the two encoding branches in the differential dual-branch encoder. Then, mid-level fusion on the saliency features of the multi-modal images is conducted in the fusion stage. Finally, the long-range lateral connections are adopted to transmit shallow features of the differential dual-branch encoder implemented to the decoder and guide the fusion process and the image reconstruction simultaneously. Experimental results show the proposed method enhances the detailed information of the fused images and achieves better performance in both visual effect and objective evaluation.

Key words： Image Fusion Infrared Image Visible Image Convolutional Neural Network

收稿日期: 2022-04-27

ZTFLH:

TN 911.73

基金资助:国家自然科学基金项目(No.62020106012,U1836218,62106089)、教育部111项目(No.B12018)资助

通讯作者: 吴小俊,博士,教授,主要研究方向为人工智能、模式识别、计算机视觉.E-mail:xiaojun_wu_jnu@163.com.

作者简介: 洪雨露,硕士研究生,主要研究方向为图像融合、深度学习.E-mail:yulu_hong@163.com.
徐天阳,博士,副教授,主要研究方向为人工智能、模式识别、计算机视觉.E-mail:tianyang_xu@163.com.

引用本文:

洪雨露, 吴小俊, 徐天阳. 基于差异双分支编码器的多阶段图像融合方法[J]. 模式识别与人工智能, 2022, 35(7): 661-670. HONG Yulu, WU Xiaojun, XU Tianyang. Multi-stage Image Fusion Method Based on Differential Dual-Branch Encoder. Pattern Recognition and Artificial Intelligence, 2022, 35(7): 661-670.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202207008 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2022/V35/I7/661

[1] MA J Y, MA Y, LI C. Infrared and Visible Image Fusion Methods and Applications: A Survey. Information Fusion, 2019, 45: 153-178.
[2] LAHMYED R, EL ANSARI M, ELLAHYANI A. A New Thermal Infrared and Visible Spectrum Images-Based Pedestrian Detection System. Multimedia Tools and Applications, 2019, 78(12): 15861-15885.
[3] 赵迪,徐志胜.基于MRSVD红外热像融合的混凝土结构火灾损伤检测方法.信息与控制, 2017, 46(1): 19-24, 40.
(ZHAO D, XU Z S. Detection of Fire Damage to Concrete Structures with Infrared Thermal Fusion Based on Multi-resolution Singular Value Decomposition. Information and Control, 2017, 46(1): 19-24, 40.)
[4] 李盼盼,王朝立,孙占全.基于注意力机制的多特征融合人脸活体检测.信息与控制, 2021, 50(5): 631-640.
(LI P P, WANG C L, SUN Z Q. Face Liveness Detection Based on Multi-feature Fusion with an Attention Mechanism. Information and Control, 2021, 50(5): 631-640.)
[5] LI C L, LIANG X Y, LU Y J, et al. RGB-T Object Tracking: Benchmark and Baseline. Pattern Recognition, 2019, 96. DOI: 10.1016/j.patcog.2019.106977.
[6] 汤张泳,吴小俊,朱学峰.多空间分辨率自适应特征融合的相关滤波目标跟踪算法.模式识别与人工智能, 2020, 33(1): 66-74.
(TANG Z Y, WU X J, ZHU X F. Object Tracking with Multi-spatial Resolutions and Adaptive Feature Fusion Based on Correlation Filters. Pattern Recognition and Artificial Intelligence, 2020, 33(1): 66-74.)
[7] 申晓霞,张桦,高赞,等.基于深度信息和RGB图像的行为识别算法.模式识别与人工智能, 2013, 26(8): 722-728.
(SHEN X X, ZHANG H, GAO Z, et al. Behavior Recognition Algo-rithm Based on Depth Information and RGB Image. Pattern Re-cognition and Artificial Intelligence, 2013, 26(8): 722-728.)
[8] BEN HAMZA A, HE Y, KRIM H, et al. A Multiscale Approach to Pixel-Level Image Fusion. Integrated Computer-Aided Engineering, 2005, 12(2): 135-146.
[9] LIU C H, QI Y, DING W R. Infrared and Visible Image Fusion Method Based on Saliency Detection in Sparse Domain. Infrared Physics & Technology, 2017, 83: 94-102.
[10] GAO R, VOROBYOV S A, ZHAO H. Image Fusion with Cosparse Analysis Operator. IEEE Signal Processing Letters, 2017, 24(7): 943-947.
[11] LI H, WU X J, KITTLER J. Infrared and Visible Image Fusion Using a Deep Learning Framework // Proc of the 24th International Conference on Pattern Recognition. Washington, USA: IEEE, 2018: 2705-2710.
[12] LI H, WU X J. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[13] LI H, WU X J, DURRANI T. NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[14] PRABHAKAR K R, SRIKAR V S, BABU R V. DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4724-4732.
[15] 肖儿良,林化溪,简献忠.基于生成对抗网络探索潜在空间的医学图像融合算法.信息与控制, 2021, 50(5): 538-549.
(XIAO E L, LIN H X, JIAN X Z. Medical Image Fusion Algorithm Adopting Generative Adversarial Network to Explore Latent Space. Information and Control, 2021, 50(5): 538-549.)
[16] MA J Y, YU W, LIANG P W, et al. FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion. Information Fusion, 2019, 48: 11-26.
[17] MA J Y, LIANG P W, YU W, et al. Infrared and Visible Image Fusion via Detail Preserving Adversarial Learning. Information Fusion, 2020, 54: 85-98.
[18] ZHANG Y, LIU Y, SUN P, et al. IFCNN: A General Image Fusion Framework Based on Convolutional Neural Network. Information Fusion, 2020, 54: 99-118.
[19] ZHANG H, XU H, XIAO Y, et al. Rethinking the Image Fusion: A Fast Unified Image Fusion Network Based on Proportional Maintenance of Gradient and Intensity. Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020, 34(7): 12797-12804.
[20] 程春阳,吴小俊,徐天阳.基于GhostNet的端到端红外和可见光图像融合方法.模式识别与人工智能, 2021, 34(11): 1028-1037.
(CHENG C Y, WU X J, XU T Y. End-to-End Infrared and Visible Image Fusion Method Based on GhostNet. Pattern Recognition and Artificial Intelligence, 2021, 34(11): 1028-1037.)
[21] XU H, MA J Y, JIANG J J, et al. U2Fusion: A Unified Unsupervised Image Fusion Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518.
[22] WANG Z, SIMONCELLI E P, BOVIK A C. Multiscale Structural Similarity for Image Quality Assessment // Proc of the 37th Asilomar Conference on Signals, Systems & Computers. Washington, USA: IEEE, 2003: 1398-1402.
[23] KRISTAN M, LEONARDIS A, MATAS J, et al. The Eighth Visual Object Tracking VOT2020 Challenge Results // Proc of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 547-601.
[24] ROBERTS J W, VAN AARDT J A, AHMED F B. Assessment of Image Fusion Procedures Using Entropy, Image Quality, and Multispectral Classification. Journal of Applied Remote Sensing, 2008, 2(1). DOI: 10.1117/1.2945910.
[25] SHEIKH H R, BOVIK A C. Image Information and Visual Quality. IEEE Transactions on Image Processing, 2006, 15(2): 430-444.
[26] QU G H, ZHANG D L, YAN P F. Information Measure for Performance of Image Fusion. Electronics Letters, 2002, 38(7): 313-315.
[27] HAGHIGHAT M, RAZIAN M A. Fast-FMI: Non-reference Image Fusion Metric // Proc of the 8th IEEE International Conference on Application of Information and Communication Technologies. Wa-shington, USA: IEEE, 2014. DOI: 10.1109/ICAICT.2014.7036000.
[28] XYDEAS C S, PETROVIC V. Objective Image Fusion Perfor-mance Measure. Electronics Letters, 2000, 36(4): 308-309.
[29] HWANG S, PARK J, KIM N, et al. Multispectral Pedestrian Detection: Benchmark Dataset and Baseline // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 1037-1045.