基于GhostNet的端到端红外和可见光图像融合方法

doi:10.16451/j.cnki.issn1003-6059.202111006

摘要
图/表
参考文献(0)
相关文章 (15)

全文: PDF (1948 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

现有的基于深度学习的红外和可见光图像融合方法大多基于人工设计的融合策略,难以为复杂的源图像设计一个合适的融合策略.针对上述问题,文中提出基于GhostNet的端到端红外和可见光图像融合方法.在网络结构中使用Ghost模块代替卷积层,形成一个轻量级模型.损失函数的约束使网络学习到适应融合任务的图像特征,从而在特征提取的同时完成融合任务.此外,在损失函数中引入感知损失,将图像的深层语义信息应用到融合过程中.源图像通过级联输入深度网络,在经过带有稠密连接的编码器提取图像特征后,通过解码器的重构得到融合结果.实验表明,文中方法在主观对比和客观图像质量评价上都有较好表现.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	程春阳
	吴小俊
	徐天阳

关键词 ：红外图像, 可见光图像, 图像融合, 深度学习

Abstract：

Most of the existing deep learning based infrared and visible image fusion methods are grounded on manual fusion strategies in the fusion layer. Therefore ,they are incapable of designing an appropriate fusion strategy for the specific image fusion task.To overcome this problem, an end-to-end infrared and visible image fusion method based on GhostNet is proposed.The Ghost module is employed to replace the ordinary convolution layer in the network architecture, and thus it becomes a lightweight model.The constraint of the loss function makes the network learn the adaptive image features for the fusion task, and consequently the feature extraction and fusion are accomplished at the same time. In addition, the perceptual loss is introduced into the design of the loss function. The deep semantic information of source images is utilized in the image fusion as well.Source images are concatenated in the channel dimension and then fed into the deep network.A densely connected encoder is applied to extract deep features of source images. The fusion result is obtained through the reconstruction of the decoder. Experiments show that the proposed method is superior in subjective comparison and objective image quality evaluation metrics.

Key words： Infrared Image Visible Image Image Fusion Deep Learning

收稿日期: 2021-05-17

ZTFLH:

TN 911.73

基金资助:

国家自然科学基金项目(No.62020106012,U1836218,616722
65)、中国教育部111项目(No.B12018)资助

通讯作者: 吴小俊,博士,教授,主要研究方向为人工智能、模式识别、计算机视觉.E-mail:xiaojun_wu_jnu@163.com.

作者简介: 程春阳,硕士研究生,主要研究方向为图像融合、深度学习.E-mail:chunyang_cheng@163.com.
徐天阳,博士,副教授,主要研究方向为人工智能、模式识别、计算机视觉.E-mail:tianyang_xu@163.com.

引用本文:

程春阳, 吴小俊, 徐天阳. 基于GhostNet的端到端红外和可见光图像融合方法[J]. 模式识别与人工智能, 2021, 34(11): 1028-1037. CHENG Chunyang, WU Xiaojun, XU Tianyang. End-to-End Infrared and Visible Image Fusion Method Based on GhostNet. , 2021, 34(11): 1028-1037.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202111006 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2021/V34/I11/1028

[1] LI S T, KANG X D, FANG L Y, et al. Pixel-Level Image Fusion: A Survey of the State of the Art. Information Fusion, 2017, 33: 100-112.
[2] ZHANG X C. Deep Learning-Based Multi-focus Image Fusion: A Survey and a Comparative Study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. DOI: 10.1109/TPAMI.2021.3078906.
[3] ZHANG H, XU H, TIAN X, et al. Image Fusion Meets Deep Learning: A Survey and Perspective. Information Fusion, 2021, 76: 323-336.
[4] 罗丽,袁真,王珂.基于àtrous-Contourlet变换的遥感图像融合算法.模式识别与人工智能, 2007, 20(2): 248-253.
(LUO L, YUAN Z, WANG K. A Remote Sensing Image Fusion Algorithm Based on àtrous-Contourlet Transformation. Pattern Reco-gnition and Artificial Intelligence, 2007, 20(2): 248-253.)
[5] BAVIRISETTI D P, DHULI R. Two-Scale Image Fusion of Visible and Infrared Images Using Saliency Detection. Infrared Physics and Technology, 2016, 76: 52-64.
[6] WANG L, LI B, TIAN L F. EGGDD: An Explicit Dependency Model for Multi-modal Medical Image Fusion in Shift-Invariant Shearlet Transform Domain. Information Fusion, 2014, 19: 29-37.
[7] 刘哲,宋余庆,陈健美,等.基于多分辨率的非参数正交多项式医学图像融合方法.模式识别与人工智能, 2012, 25(2): 300-304.
(LIU Z, SONG Q Y, CHEN J M, et al. A Multi-resolution Medical Image Fusion Method Based on Nonparametric Orthogonal Polyno-mials. Pattern Recognition and Artificial Intelligence, 2012, 25(2): 300-304.)
[8] LIU C H, QI Y, DING W R, et al. Infrared and Visible Image Fusion Method Based on Saliency Detection in Sparse Domain. Infrared Physics and Technology, 2017, 83: 94-102.
[9] LI H, WU X J. Multi-focus Image Fusion Using Dictionary Learning and Low-Rank Representation // Proc of the International Confe-rence on Image and Graphics. Berlin, Germany: Springer, 2017: 675-686.
[10] LIU Y, CHEN X, PENG H, et al. Multi-focus Image Fusion with a Deep Convolutional Neural Network. Information Fusion, 2017, 36: 191-207.
[11] MA J Y, YU W, LIANG P W, et al. FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion. Information Fusion, 2019, 48: 11-26.
[12] PRABHAKAR R K, SRIKAR V S, BABU R V. DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4724-4732.
[13] LI H, WU X J. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[14] XU H, MA J Y, JIANG J J, et al. U2Fusion: A Unified Unsupervised Image Fusion Network. IEEE Transactions on Pattern Analy-sis and Machine Intelligence, 2020. DOI: 10.1109/TPAMI.2020.3012548.
[15] ZHANG H, XU H, XIAO Y, et al. Rethinking the Image Fusion: A Fast Unified Image Fusion Network Based on Proportional Maintenance of Gradient and Intensity. Proceedings of the AAAI Confe-rence on Artificial Intelligence, 2020, 34(7): 12797-12804.
[16] LI H, WU X J, DURRANI T. NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[17] HAN K, WANG Y H, TIAN Q, et al. GhostNet: More Features from Cheap Operations // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1577-1586.
[18] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269.
[19] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[20] ZHANG Y, LIU Y, SUN P, et al. IFCNN: A General Image Fusion Framework Based on Convolutional Neural Network. Information Fusion, 2020, 54: 99-118.
[21] ZHOU W, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[22] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-04-26]. https://arxiv.org/pdf/1409.1556v6.pdf.
[23] MA J Y, CHEN C, LI C, et al. Infrared and Visible Image Fusion via Gradient Transfer and Total Variation Minimization. Information Fusion, 2016, 31: 100-109.
[24] ASLANTAS V, BENDES E. A New Image Quality Metric for Image Fusion: The Sum of the Correlations of Differences. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890-1896.
[25] HAN Y, CAI Y Z, CAO Y, et al. A New Image Fusion Perfor-
mance Metric Based on Visual Information Fidelity. Information Fusion, 2013, 14(2): 127-135.
[26] LI H, WU X J, KITTLER J. RFN-Nest: An End-to-End Residual Fusion Network for Infrared and Visible Images. Information Fusion, 2021, 73: 72-86.