End-to-End Infrared and Visible Image Fusion Method Based on GhostNet
CHENG Chunyang1, WU Xiaojun1, XU Tianyang1
1. Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122
Most of the existing deep learning based infrared and visible image fusion methods are grounded on manual fusion strategies in the fusion layer. Therefore ,they are incapable of designing an appropriate fusion strategy for the specific image fusion task.To overcome this problem, an end-to-end infrared and visible image fusion method based on GhostNet is proposed.The Ghost module is employed to replace the ordinary convolution layer in the network architecture, and thus it becomes a lightweight model.The constraint of the loss function makes the network learn the adaptive image features for the fusion task, and consequently the feature extraction and fusion are accomplished at the same time. In addition, the perceptual loss is introduced into the design of the loss function. The deep semantic information of source images is utilized in the image fusion as well.Source images are concatenated in the channel dimension and then fed into the deep network.A densely connected encoder is applied to extract deep features of source images. The fusion result is obtained through the reconstruction of the decoder. Experiments show that the proposed method is superior in subjective comparison and objective image quality evaluation metrics.
[1] LI S T, KANG X D, FANG L Y, et al. Pixel-Level Image Fusion: A Survey of the State of the Art. Information Fusion, 2017, 33: 100-112.
[2] ZHANG X C. Deep Learning-Based Multi-focus Image Fusion: A Survey and a Comparative Study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. DOI: 10.1109/TPAMI.2021.3078906.
[3] ZHANG H, XU H, TIAN X, et al. Image Fusion Meets Deep Learning: A Survey and Perspective. Information Fusion, 2021, 76: 323-336.
[4] 罗 丽,袁 真,王 珂.基于àtrous-Contourlet变换的遥感图像融合算法.模式识别与人工智能, 2007, 20(2): 248-253.
(LUO L, YUAN Z, WANG K. A Remote Sensing Image Fusion Algorithm Based on àtrous-Contourlet Transformation. Pattern Reco-gnition and Artificial Intelligence, 2007, 20(2): 248-253.)
[5] BAVIRISETTI D P, DHULI R. Two-Scale Image Fusion of Visible and Infrared Images Using Saliency Detection. Infrared Physics and Technology, 2016, 76: 52-64.
[6] WANG L, LI B, TIAN L F. EGGDD: An Explicit Dependency Model for Multi-modal Medical Image Fusion in Shift-Invariant Shearlet Transform Domain. Information Fusion, 2014, 19: 29-37.
[7] 刘 哲,宋余庆,陈健美,等.基于多分辨率的非参数正交多项式医学图像融合方法.模式识别与人工智能, 2012, 25(2): 300-304.
(LIU Z, SONG Q Y, CHEN J M, et al. A Multi-resolution Medical Image Fusion Method Based on Nonparametric Orthogonal Polyno-mials. Pattern Recognition and Artificial Intelligence, 2012, 25(2): 300-304.)
[8] LIU C H, QI Y, DING W R, et al. Infrared and Visible Image Fusion Method Based on Saliency Detection in Sparse Domain. Infrared Physics and Technology, 2017, 83: 94-102.
[9] LI H, WU X J. Multi-focus Image Fusion Using Dictionary Learning and Low-Rank Representation // Proc of the International Confe-rence on Image and Graphics. Berlin, Germany: Springer, 2017: 675-686.
[10] LIU Y, CHEN X, PENG H, et al. Multi-focus Image Fusion with a Deep Convolutional Neural Network. Information Fusion, 2017, 36: 191-207.
[11] MA J Y, YU W, LIANG P W, et al. FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion. Information Fusion, 2019, 48: 11-26.
[12] PRABHAKAR R K, SRIKAR V S, BABU R V. DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4724-4732.
[13] LI H, WU X J. DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[14] XU H, MA J Y, JIANG J J, et al. U2Fusion: A Unified Unsupervised Image Fusion Network. IEEE Transactions on Pattern Analy-sis and Machine Intelligence, 2020. DOI: 10.1109/TPAMI.2020.3012548.
[15] ZHANG H, XU H, XIAO Y, et al. Rethinking the Image Fusion: A Fast Unified Image Fusion Network Based on Proportional Maintenance of Gradient and Intensity. Proceedings of the AAAI Confe-rence on Artificial Intelligence, 2020, 34(7): 12797-12804.
[16] LI H, WU X J, DURRANI T. NestFuse: An Infrared and Visible Image Fusion Architecture Based on Nest Connection and Spatial/Channel Attention Models. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[17] HAN K, WANG Y H, TIAN Q, et al. GhostNet: More Features from Cheap Operations // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 1577-1586.
[18] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely Connected Convolutional Networks // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 2261-2269.
[19] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[20] ZHANG Y, LIU Y, SUN P, et al. IFCNN: A General Image Fusion Framework Based on Convolutional Neural Network. Information Fusion, 2020, 54: 99-118.
[21] ZHOU W, BOVIK A C, SHEIKH H R, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[22] SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[C/OL]. [2021-04-26]. https://arxiv.org/pdf/1409.1556v6.pdf.
[23] MA J Y, CHEN C, LI C, et al. Infrared and Visible Image Fusion via Gradient Transfer and Total Variation Minimization. Information Fusion, 2016, 31: 100-109.
[24] ASLANTAS V, BENDES E. A New Image Quality Metric for Image Fusion: The Sum of the Correlations of Differences. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890-1896.
[25] HAN Y, CAI Y Z, CAO Y, et al. A New Image Fusion Perfor-
mance Metric Based on Visual Information Fidelity. Information Fusion, 2013, 14(2): 127-135.
[26] LI H, WU X J, KITTLER J. RFN-Nest: An End-to-End Residual Fusion Network for Infrared and Visible Images. Information Fusion, 2021, 73: 72-86.
2021, Vol. 34 
