基于鸟瞰图的空间-通道注意力多传感器融合

doi:10.16451/j.cnki.issn1003-6059.202311006

摘要
图/表
参考文献(0)
相关文章 (15)

全文: PDF (1983 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要

基于鸟瞰图(Bird's-Eye View, BEV)的目标感知算法现正成为研究热点,但少有针对BEV的多传感器融合.因此,文中提出基于空间-通道注意力的多传感器融合模块,对不同模态的特征数据增加局部注意力机制,能有效修正多传感器之间的空间误差.利用转置注意力操作,充分融合图像和点云数据,消解不同模态语义信息之间的异质性,使融合的BEV特征在不引入空间偏差的同时,有效结合多传感器各自的独特信息,实现更全面准确的感知.在nuScenes数据集上的实验以及大量的消融实验表明,文中融合模块能有效提升目标检测算法的精度,可视化结果展现融合后的特征具有更完整、准确的特征信息,可明显提升对远处物体的检测.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	吉宇哲
	陈奕洁
	杨柳青
	郑心湖

关键词 ：鸟瞰图(BEV), 多传感器融合, 注意力机制, 目标检测

Abstract：

Object perception based on bird's-eye view(BEV) is one of hot issues, but studies on multi-sensor fusion for BEV are still insufficient. Therefore, a multi-sensor fusion module based on spatial-channel attention is proposed. Spatial errors between multiple sensors can be effectively corrected by adding local attention mechanisms to features of different modalities. By using transpose attention operations, the image and point cloud data are fully integrated to resolve the heterogeneity between different modal semantics. Consequently, the fused BEV features achieves more comprehensive and accurate perception by effectively combining the unique information of each sensor without introducing spatial misalignment. Experiment on nuScenes dataset and extensive ablation experiments show that the proposed fusion module effectively improves the accuracy of object detection. Visualization results demonstrate that the fused features can capture more complete and accurate information, especially in distant objects detection.

Key words： Bird's-Eye View(BEV) Multi-sensor Fusion Attention Mechanism Object Detection

收稿日期: 2023-10-18

ZTFLH:

TP391.41

基金资助:

国家自然科学基金面上项目(No.62373315)、广州市科技计划项目(No.2023A03J0683,2023A03J0011)资助

作者简介: 吉宇哲,博士研究生,主要研究方向为多模态融合感知、多车协同感知.E-mail:yji755@connect.hkust-gz.edu.cn.陈奕洁,硕士研究生,主要研究方向为多智能体协同感知、车路协同系统.E-mail:ychen324@connect.hkust-gz.edu.cn.杨柳青,博士,教授,主要研究方向为无线通信网络、多智能体系统、通讯感知一体化等.E-mail:lqyang@ust.hk.郑心湖,博士,助理教授,主要研究方向为多模态感知、多智能体协同感知、网联智能.E-mail:xinhuzheng@ust.hk.

引用本文:

吉宇哲, 陈奕洁, 杨柳青, 郑心湖. 基于鸟瞰图的空间-通道注意力多传感器融合[J]. 模式识别与人工智能, 2023, 36(11): 1029-1040. JI Yuzhe, CHEN Yijie, YANG Liuqing, ZHENG Xinhu. Spatial-Channel Attention Multi-sensor Fusion Based on Bird's-Eye View. Pattern Recognition and Artificial Intelligence, 2023, 36(11): 1029-1040.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202311006 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2023/V36/I11/1029

[1] IOANNIDOU A, CHATZILARI E, NIKOLOPOULOS S, et al. Deep Learning Advances in Computer Vision with 3D Data: A Survey. ACM Computing Surveys, 2018, 50(2). DOI: 10.1145/3042064.
[2] HUANG K, SHI B T, LI X, et al. Multi-modal Sensor Fusion for Auto Driving Perception: A Survey[C/OL].[2023-08-12]. https://arxiv.org/pdf/2202.02703v1.pdf.
[3] FENG D, HAASE-SCHÜTZ C, ROSENBAUM L, et al. Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges. IEEE Transactions on Intelligent Transportation Systems, 2020, 22(3): 1341-1360.
[4] FAYYAD J, JARADAT M A, GRUYER D, et al. Deep Learning Sensor Fusion for Autonomous Vehicle Perception and Localization: A Review. Sensors, 2020, 20(15). DOI: 10.3390/s20154220.
[5] PANG S, MORRIS D, RADHA H.CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection // Proc of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Washing-ton, USA: IEEE, 2020: 10386-10393.
[6] ASVADI A, GARROTE L, PREMEBIDA C, et al. Multimodal Vehicle Detection: Fusing 3D-LiDAR and Color Camera Data. Pattern Recognition Letters, 2018, 115: 20-29.
[7] VORA S, LANG A H, HELOU B, et al. Pointpainting: Sequential Fusion for 3D Object Detection // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 4603-3611.
[8] XIE L, XIANG C, YU Z X, et al. PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-Based Attentive Cont-Conv Fusion Module. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12460-12467.
[9] LIANG M, YANG B, CHEN Y, et al. Multi-task Multi-sensor Fusion for 3D Object Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2019: 7337-7345.
[10] YOU Y R, WANG Y, CHAO W L, et al. Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving[C/OL].[2023-08-12]. https://arxiv.org/pdf/1906.06310v1.pdf.
[11] WANG J D, WEI Z, ZHANG T, et al. Deeply-Fused Nets[C/OL].[2023-08-12]. https://arxiv.org/abs/1605.07716.
[12] PHILION J, FIDLER S. Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D // Proc of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 194-210.
[13] HUANG J J, HUANG G, ZHU Z, et al. BEVDet: High-Perfor-mance Multi-camera 3D Object Detection in Bird-Eye-View[C/OL].[2023-08-12]. https://arxiv.org/pdf/2112.11790v1.pdf.
[14] XIE B Z, YU Z D, ZHOU D Q, et al. M²BEV: Multi-camera Joint 3D Detection and Segmentation with Unified Birds-Eye View Representation[C/OL].[2023-08-12]. https://arxiv.org/pdf/2204.05088.pdf.
[15] ZHOU Y, TUZEL O.VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 4490-4499.
[16] YANG Y, MAO Y X, LI B Y.SECOND: Sparsely Embedded Con-volutional Detection. Sensors, 2018, 18(10). DOI: 10.3390/s18103337.
[17] ZHOU Y, SUN P, ZHANG Y, et al. End-to-End Multi-view Fusion for 3D Object Detection in LiDAR Point Clouds // Proc of the Conference on Robot Learning. San Diego, USA: JMLR, 2020: 923-932.
[18] LECUN Y, BENGIO Y. HINTON G. Deep Learning. Nature, 2015, 521: 436-444.
[19] LIU Z J, TANG H T, AMINI A, et al. BEVFusion: Multi-task Multi-sensor Fusion with Unified Bird's-Eye View Representation // Proc of the IEEE International Conference on Robotics and Automation. Washington, USA: IEEE, 2023: 2774-2781.
[20] BORSE S, KLINGNER M, KUMAR V R, et al. X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation // Proc of the IEEE/CVF Winter Conference on Applications of Computer Vision. Washington, USA: IEEE, 2023: 3286-3296.
[21] LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 9992-10002.
[22] LIN T Y, DOLLÁR R, GIRSHICK K, et al. Feature Pyramid Networks for Object Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 936-944.
[23] BAI X Y, HU Z Y, ZHU X G, et al. Transfusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2022: 1080-1089.
[24] SZEGEDY C, LIU W, JIA Y Q, et al. Going Deeper with Convolutions // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015. DOI: 10.1109/CVPR.2015.7298594.
[25] SZEGEDY C, VANHOUCKE V, LOFFE S, et al. Rethinking the Inception Architecture for Computer Vision // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 2818-2826.
[26] ZHANG H, WU C R, ZHANG Z Y, et al. ResNeSt: Split-Attention Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2022: 2735-2745.
[27] EL-NOUBY A, TOUVRON H, CARON M, et al. XCiT: Cross-Covariance Image Transformers[C/OL].[2023-08-12]. https://arxiv.org/pdf/2106.09681.pdf.
[28] CAESAR H, BANKITI V, LANG A H, et al. nuScenes: A Multimodal Dataset for Autonomous Driving // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 11618-11628.
[29] GEIGER A, URTASUN R.Vision Meets Robotics: The KITTI Da-taset. The International Journal of Robotics Research, 2013, 32(11): 1231-1237.
[30] LOSHCHILOV L, HUTTER F.Decoupled Weight Decay Regularization[C/OL]. [2023-08-12].https://arxiv.org/abs/1711.05101.