|
|
|
| A Review of Deep Learning-Based Signal Fusion Methods for Multimodal Brain-Computer Interfaces |
| LI Jinhai1,2, LIU Siyi1,2, SHI Shuiling1,2, LIU Wenqi1,2 |
1. Yunnan Key Laboratory of Complex Systems and Brain-Inspired Intelligence, Kunming University of Science and Technology, Kunming 650500;
2. Faculty of Science, Kunming University of Science and Technology, Kunming 650500 |
|
|
|
|
Abstract The signal fusion of multimodal brain-computer interfaces significantly improves decoding accuracy and robustness by mitigating the inherent drawbacks of a single modality in spatiotemporal resolution and signal-to-noise ratio. It is considered as an effective method for improving the performance of brain-computer interfaces. This paper is intended to synthesize the existing research results of deep learning-based multimodal brain-computer interfaces. First, existing literature is systematically classified into four categories based on different fusion strategies for modal data: data-level fusion, feature-level fusion, decision-level fusion, and hybrid strategy fusion. Then, based on the characteristics of deep learning models, fusion strategies are further subdivided and analyzed. The application scenarios of multimodal brain-computer interfaces are summarized, including the fields of motor imagery and motor rehabilitation, assessment and classification of the cognitive load, vigilance monitoring, emotion recognition,and auxiliary diagnosis and functional evaluation of neurological diseases. Finally, the difficulties faced by current signal fusion methods and multimodal deep learning are summarized. The future research directions are discussed and the future prospects are outlined.
|
|
Received: 12 May 2026
|
|
|
| Fund:National Natural Science Foundation of China(No.62476114,12371460), Yunnan Fundamental Research Projects(No.202401AV070009) |
|
Corresponding Authors:
LI Jinhai, Ph.D., professor. His research interests include cognitive computing, granular computing, big data analysis, concept lattice and rough set.
|
| About author:: About Author:
LIU Siyi, Ph.D.candidate. Her research interests include brain-computer interface, multi-modal fusion and concept lattice.
SHI Shuiling, Ph.D.candidate. Her research interests include machine learning, broad learning system, computational intelligence, affective computing and complex networks.
LIU Wenqi, Master, professor. His research interests include data governance, brain and cognitive complex systems.Academic Papers of the 28th Annual Meeting of the China Association for Science and Technology |
|
|
|
[1] PRESIGNY C, DE VICO FALLANI F. Colloquium: Multiscale Mo-deling of Brain Network Organization. Reviews of Modern Physics, 2022, 94. DOI: 10.1103/RevModPhys.94.031002.
[2] MA C, LI W L, GAO C, et al. Multifunctional Hydrogel Materials for Advanced Neural Interfaces. Small Methods, 2025, 9(9). DOI: 10.1002/smtd.202501134.
[3] LI D, LI R Y, SONG Y P, et al. Effects of Brain-Computer Interface Based Training on Post-Stroke Upper-Limb Rehabilitation: A Meta-Analysis. Journal of Neuroengineering and Rehabilitation, 2025, 22(1). DOI: 10.1186/s12984-025-01588-x
[4] WANG S Q, GUO Y, DONG Y H, et al. Generative AI Empowers Brain-Computer Interfaces: A Review-Perspective on Technical Realities and Future Visions. IEEE Transactions on Consumer Electronics, 2026, 72(1): 11-20.
[5] MA Y N, KARAKO K, SONG P P, et al. Integrative Neurorehabilitation Using Brain-Computer Interface: From Motor Function to Mental Health after Stroke. Bioscience Trends, 2025, 19(3): 243-251.
[6] ZHU J B, LI K D, CHEN S C, et al. Smart Ward Control Based on a Wearable Multimodal Brain-Computer Interface Mouse. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2026, 34: 638-649.
[7] GOUDA S K, CHOUDHRY A, SATPATHY S P, et al. Integration of EEG-Based BCI Technology in IoT Enabled Smart Home Environment: An In-Depth Comparative Analysis on Human-Computer Interaction Techniques. Expert Systems with Applications, 2025, 294. DOI: 10.1016/j.eswa.2025.128730.
[8] LIU S Y, LIU M Z, ZHANG D Y, et al. Brain-Controlled Hand Exoskeleton Based on Augmented Reality-Fused Stimulus Paradigm. IEEE Journal of Biomedical and Health Informatics, 2025, 29(4): 2932-2944.
[9] ZHANG H Z, TAO C J.Deep Learning Decoding of Steady-State Visual Evoked Potential(SSVEP) for Real-Time Mobile Brain-Computer Interfaces: A Narrative Review from Laboratory Settings to Lightweight Engineering Applications. Brain Sciences, 2026, 16(4). DOI: 10.3390/brainsci16040387.
[10] THAKUR S, THAKUR S, RANA A,et al. A Systematic Review of Techniques, Artifacts, Challenges and Future Directions on Signal Acquisition and Preprocessing in Brain-Computer Interfaces(BCIs). International Journal of Human-Computer Interaction, 2026. DOI: 10.1080/10447318.2025.2602691.
[11] KHAN H, NASEER N, YAZIDI A, et al. Analysis of Human Gait Using Hybrid EEG-fNIRS-Based BCI System: A Review. Frontiers in Human Neuroscience, 2021, 14. DOI: 10.3389/fnhum.2020.613254.
[12] ELMAN J L.Finding Structure in Time. Cognitive Science, 1990, 14(2): 179-211.
[13] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-Based Lear-ning Applied to Document Recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
[14] LEA C, FLYNN M D, VIDAL R, et al. Temporal Convolutional Networks for Action Segmentation and Detection // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2017: 1003-1012.
[15] HOCHREITER S, SCHMIDHUBER J.Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780.
[16] GRAVES A, SCHMIDHUBER J.Framewise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures. Neural Networks, 2005, 18(5/6): 602-610.
[17] CHO K, VAN MERRIENBOER B, GULÇEHRE Ç , et al. Lear-ning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2014: 1724-1734.
[18] KIPF T N, WELLING M.Semi-Supervised Classification with Graph Convolutional Networks [C/OL]. [2026-04-12]. https://arxiv.org/pdf/1609.02907.
[19] VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph Attention Networks[C/OL].[2026-04-12]. https://arxiv.org/pdf/1710.10903.
[20] RAGESH R, SELLAMANICKAM S, IYER A, et al. HeteGCN: Heterogeneous Graph Convolutional Networks for Text Classification // Proc of the 14th ACM International Conference on Web Search and Data Mining. New York, USA: ACM, 2021: 860-868.
[21] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010.
[22] ZHANG X, WU D R, DING L Y, et al. Tiny Noise, Big Mistakes: Adversarial Perturbations Induce Errors in Brain-Computer Interface Spellers. National Science Review, 2021, 8(4). DOI: 10.1093/nsr/nwaa233.
[23] WU D R, LU B L, HU B, et al. Affective Brain-Computer Interfaces(aBCIs): A Tutorial. Proceedings of the IEEE, 2023, 111(10): 1314-1332.
[24] O'DOHERTY J E, LEBEDEV M A, IFFT P J, et al. Active Tac-tile Exploration Using a Brain-Machine-Brain Interface. Nature, 2011, 479(7372): 228-231.
[25] MANE R, CHOUHAN T, GUAN C T.BCI for Stroke Rehabilitation: Motor and Beyond. Journal of Neural Engineering, 2020, 17(4). DOI: 10.1088/1741-2552/aba162.
[26] EDELMAN B J, MENG J, SUMA D, et al. Noninvasive Neuroi-maging Enhances Continuous Neural Tracking for Robotic Device Control. Science Robotics, 2019, 4(31). DOI: 10.1126/scirobotics.aaw6844.
[27] MOSES D A, METZGER S L, LIU J R, et al. Neuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria. The New England Journal of Medicine, 2021, 385(3): 217-227.
[28] LORACH H, GALVEZ A, SPAGNOLO V, et al. Walking Naturally after Spinal Cord Injury Using a Brain-Spine Interface. Nature, 2023, 618(7963): 126-133.
[29] BAR A, LOHDEFINK J, KAPOOR N, et al. The Vulnerability of Semantic Segmentation Networks to Adversarial Attacks in Autonomous Driving: Enhancing Extensive Environment Sensing. IEEE Signal Processing Magazine, 2021, 38(1): 42-52.
[30] LI Y Q, PAN J H, LONG J Y, et al. Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients with Disorder of Consciousness. Proceedings of the IEEE, 2015, 104(2): 332-352.
[31] OTARBAY Z, KYZYRKANOV A.SVM-Enhanced Attention Me-chanisms for Motor Imagery EEG Classification in Brain-Computer Interfaces. Frontiers in Neuroscience, 2025, 19. DOI: 10.3389/fnins.2025.1622847.
[32] YAO R W, DU Z H, LIANG F S, et al. Dual-Layer Hand Gestures Decoding with Wireless Epidural Braincomputer Interface in a Tetraplegia // Proc of the 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2025. DOI: 10.1109/EMBC58623.2025.11254206.
[33] LIM M J R, LO J Y T, TAN Y Y, et al. The State-of-the-Art of Invasive Brain-Computer Interfaces in Humans: A Systematic Review and Individual Patient Meta-Analysis. Journal of Neural Engineering, 2025, 22(2). DOI: 10.1088/1741-2552/adb88e.
[34] ZHANG X, WU D R.On the Vulnerability of CNN Classifiers in EEG-Based BCIs. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 27(5): 814-825.
[35] ZHENG W L, LU B L. A Multimodal Approach to Estimating Vigilance Using EEG and Forehead EOG. Journal of Neural Enginee-ring, 2017, 14(2). DOI: 10.1088/1741-2552/aa5a98.
[36] BRUNNER C, LEEB R, MÜLLER-PUTZ G, ,et al. BCI Competition 2008-Graz Data Set A. Institute for Knowledge Discovery [C/OL]. [2026-04-12]. https://lampx.tugraz.at/~bci/database/001-2014/description.pdf.
[37] MARKOVA V, GANCHEV T, KALINKOV K.CLAS: A Database for Cognitive Load, Affect and Stress Recognition // Proc of the International Conference on Biomedical Innovations and Applications. Washington, USA: IEEE, 2019. DOI: 10.1109/BIA48344.2019.8967457.
[38] MCCLARY C, NASER Y M M, REPOLE S, et al. Dataset of Multi-modal Physiological Signals(fNIRS EEG ECG EMG) Recorded across Different Emotional States[C/OL].[2026-04-12]. https://ieee-dataport.org/documents/dataset-multi-modal-physiological-signals-fnirs-eeg-ecg-emg-recorded-across-different.
[39] YANG C T, PARISH O, KLIMOVICH-GRAY A, et al. Kymata Soto Language Dataset: An Electro-Magnetoencephalographic Dataset for Natural Speech Processing. Scientific Data, 2026, 13. DOI: 10.1038/s41597-026-06579-8.
[40] KOELSTRA S, MUHL C, SOLEYMANI M, et al. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Transactions on Affective Computing, 2012, 3(1): 18-31.
[41] KATSIGIANNIS S, RAMZAN N.DREAMER: A Database for Emo-tion Recognition through EEG and ECG Signals from Wireless Low-Cost Off-the-Shelf Devices. IEEE Journal of Biomedical and Health Informatics, 2018, 22(1): 98-107.
[42] VAGAJA K, VOURVOPOULOS A.Electrophysiological Signals of Embodiment and MI-BCI Training in VR[C/OL]. [2026-04-12].https://zenodo.org/records/8086086.
[43] ORTEGA P, ZHAO T, FAISAL A A.HYGRIP: Full-Stack Cha-racterization of Neurobehavioral Signals(fNIRS, EEG, EMG, For-ce, and Breathing) during a Bimanual Grip Force Control Task. Frontiers in Neuroscience, 2020, 14. DOI: 10.3389/fnins.2020.00919.
[44] BUCCINO A P, KELES H O, OMURTAG A.Hybrid EEG-fNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks. PLoS One, 2016, 11(1). DOI: 10.1371/journal.pone.0146610.
[45] LUCIW M D, JAROCKA E, EDIN B B.Multi-channel EEG Recordings during 3,936 Grasp and Lift Trials with Varying Weight and Friction. Scientific Data, 2014, 1. DOI: 10.1038/sdata.2014.47.
[46] ANGKAN P, BEHINAEIN B, MAHMUD Z, et al. Multimodal Brain-Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines. IEEE Transactions on Inte-lligent Transportation Systems, 2024, 25(6): 5949-5964.
[47] JEONG J H, CHO J H, SHIM K H, et al. Multimodal Signal Dataset for 11 Intuitive Movement Tasks from Single Upper Extre-mity during Multiple Recording Sessions. GigaScience, 2020, 9(10). DOI: 10.1093/gigascience/giaa098.
[48] SHIN J, VON LÜHMANN A, BLANKERTZ B, et al. Open Access Dataset for EEG+NIRS Single-Trial Classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(10): 1735-1745.
[49] SHIN J, VON LÜHMANN A, KIM D W, et al. Simultaneous Acquisition of EEG and NIRS During Cognitive Tasks for an Open Access Dataset. Scientific Data, 2018, 5(1). DOI: 10.1038/sdata.2018.3.
[50] SHIN J, VON LÜHMANN A, BLANKERTZ B, et al. Open Access Repository for Hybrid EEG-NIRS Data // Proc of the 6th International Conference on Brain-Computer Interface. Washington, USA: IEEE, 2018. DOI: 10.1109/IWW-BCI.2018.8311523.
[51] ZHAO S, YI S, ZHOU Y X, et al. Wearable Music2Emotion: Asse-ssing Emotions Induced by AI-Generated Music through Portable EEG-fNIRS Fusion // Proc of the 33rd ACM International Confe-rence on Multimedia. New York, USA: ACM, 2025: 5627-5636.
[52] LUO W T, AL-QANESS M A A, LI Y F, et al. EEG-Based Brain-Computer Interface: Fundamentals, Methods, Applications, and Challenges. IEEE Internet of Things Journal, 2025, 12(24): 52024-52041.
[53] LIU X Y, WANG W L, LIU M, et al. Recent Applications of EEG-Based Brain-Computer-Interface in the Medical Field. Military Medical Research, 2025, 12. DOI: 10.1186/s40779-025-00598-z.
[54] WANG Y, LIU X, CUI H Y, et al. An Auditory Selective Attention Brain-Computer Interface System Based on Auditory Steady-State Response. Applied Acoustics, 2025, 228. DOI: 10.1016/j.apacoust.2024.110291.
[55] KIM K T, LEE J, KIM H, et al. Classification of Selective Attention Based on Steady-State Somatosensory Evoked Potentials Using High-Frequency Vibration Stimuli // Proc of the 8th International Winter Conference on Brain-Computer Interface. Washington, USA: IEEE, 2020. DOI: 10.1109/BCI48061.2020.9061670.
[56] RON-ANGEVIN R, FERNÁNDEZ-RODRÍGUEZ Á, VELASCO-ÁLVAREZ F, et al. Evaluation of Different Types of Stimuli in an Event-Related Potential-Based Brain-Computer Interface Speller under Rapid Serial Visual Presentation. Sensors, 2024, 24(11). DOI: 10.3390/s24113315.
[57] REICHERT C, SWEENEY-REED C M, HINRICHS H, et al. A Toolbox for Decoding BCI Commands Based on Event-Related Potentials. Frontiers in Human Neuroscience, 2024, 18. DOI: 10.3389/fnhum.2024.1358809.
[58] CHERLOO M N, MIJANI A M, ZHAN L, et al. A Novel Multiclass-Based Framework for P300 Detection in BCI Matrix Speller: Temporal EEG Patterns of Non-Target Trials Vary Based on Their Position to Previous Target Stimuli. Engineering Applications of Artificial Intelligence, 2023, 123(B). DOI: 10.1016/j.engappai.2023.106381.
[59] WANG L, ZHANG Z H, HAN D, et al. Single Stimulus Location for Two Inputs: A Combined Brain-Computer Interface Based on Steady-State Visual Evoked Potential (SSVEP). European Journal of Neuroscience, 2021, 53(3): 861-875.
[60] LI M L, HE D N, LI C, et al. Brain-Computer Interface Speller Based on Steady-State Visual Evoked Potential: A Review Focusing on the Stimulus Paradigm and Performance. Brain Sciences, 2021, 11(4). DOI: 10.3390/brainsci11040450.
[61] CHEN Y K, SHI X Y, DE SILVA V, et al. Steady-State Visual Evoked Potential-Based Brain-Computer Interface System for Enhanced Human Activity Monitoring and Assessment. Sensors, 2024, 24(21). DOI: 10.3390/s24217084.
[62] LIU G Y, ZHANG R, TIAN L, et al. Fine-Grained Spatial-Frequency-Time Framework for Motor Imagery Brain-Computer Interface. IEEE Journal of Biomedical and Health Informatics, 2025, 29(6): 4121-4133.
[63] AZZOUGUI S, REFFAD A, MEBARKIA K.Multi-class Motor Ima-gery Classification in Brain Computer Interface(BCI) // Proc of the 2nd International Conference on Electrical Engineering and Automatic Control. Washington, USA: IEEE, 2024. DOI: 10.1109/ICEEAC61226.2024.10576368.
[64] QUARESIMA V, BISCONTI S, FERRARI M.A Brief Review on the Use of Functional Near-Infrared Spectroscopy(fNIRS) for Language Imaging Studies in Human Newborns and Adults. Brain and Language, 2012, 121(2). DOI: 10.1016/j.bandl.2011.03.009.
[65] XIE L, LIU Y, GAO Y L, et al. Functional Near-Infrared Spectroscopy in Neurodegenerative Disease: A Review. Frontiers in Neuroscience, 2024, 18. DOI: 10.3389/fnins.2024.1469903.
[66] GLOVER G H.Overview of Functional Magnetic Resonance Imaging. Neurosurgery Clinics of North America, 2011, 22(2): 133-139.
[67] YUE J H, WANG P, GUO X F, et al. Perspectives on Resting-State Functional Magnetic Resonance Imaging Research in Vascular Dementia. Frontiers in Aging Neuroscience, 2025, 17. DOI: 10.3389/fnagi.2025.1547965.
[68] OLIKKAL P, PILLAI B M, SUTHAKORN J, et al. A Hybrid EEG-EMG Framework for Humanoid Control Using Deep Learning Transformers // Proc of the IEEE International Conference on Robotics and Biomimetics. Washington, USA: IEEE, 2024: 899-904.
[69] RAMIREZ M, KHALIL M A, CAN J, et al. Classification of “Like” and “Dislike” Decisions from EEG and fNIRS Signals Using a LSTM Based Deep Learning Network // Proc of the IEEE World AI IoT Congress. Washington, USA: IEEE, 2022: 252-255.
[70] GHONCHI H, FATEH M, ABOLGHASEMI V, et al. Deep Recurrent-Convolutional Neural Network for Classification of Simultaneous EEG-fNIRS Signals. IET Signal Processing, 2020, 14(3): 142-153.
[71] CHEN J M, WANG D, HU B, et al. MCFHNet: Multi-channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding // Proc of the 44th Annual International Confe-rence of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2022: 4821-4825.
[72] HE Q, FENG L F, JIANG G Q, et al. Multimodal Multitask Neural Network for Motor Imagery Classification with EEG and fNIRS Signals. IEEE Sensors Journal, 2022, 22(21): 20695-20706.
[73] BORRA D, FRATERNALI M, RAVANELLI M,et al. Multi-modal Decoding of Reach-to-Grasping from EEG and EMG via Neural Networks // Proc of the 16th IAPRTC3 Workshop on Artificial Neural Networks in Pattern Recognition. Berlin, Germany: Sprin-ger, 2024: 168-179.
[74] NOUR M, ÖZTÜRK S , POLAT K. A Novel Classification Framework Using Multiple Bandwidth Method with Optimized CNN for Brain-Computer Interfaces with EEG-fNIRS Signals. Neural Computing and Applications, 2021, 33(22): 15815-15829.
[75] JIANG T F, GUO H, LU K Y, et al. Dual-Scale Fusion Neural Network(DFNN) for Robust Hand Rehabilitation Action Recognition under Muscle Fatigue Conditions // Proc of the IEEE International Conference on Cyborg and Bionic Systems. Washington, USA: IEEE, 2025: 365-370.
[76] MYRICK C, GHOSH I, JAYARAJAH K, et al. CALM: Multimodal Cognitive Load Assessment Framework via Engineered and Explainable Features // Proc of the IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events. Washington, USA: IEEE, 2025: 213-218.
[77] KIM S, SHIN D Y, KIM T, et al. Enhanced Recognition of Amputated Wrist and Hand Movements by Deep Learning Method Using Multimodal Fusion of Electromyography and Electroencephalography. Sensors, 2022, 22(2). DOI: 10.3390/s22020680.
[78] SAADATI M, NELSON J, AYAZ H.Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification // Proc of the AHFE International Conference on Neuroergonomics and Cognitive Engineering, and the AHFE International Conference on Industrial Cognitive Ergonomics and Engineering Psychology. Berlin, Germany: Springer, 2020: 221-232.
[79] BUNTERNGCHIT C, WANG J X, HOU Z G.Simultaneous EEG-fNIRS Data Classification through Selective Channel Representation and Spectrogram Imaging. IEEE Journal of Translational Engineering in Health and Medicine, 2024, 12: 600-612.
[80] RAMIREZ M, KAHEH S, KHALIL M A, et al. Application of Convolutional Neural Network for Classification of Consumer Prefe-rence from Hybrid EEG and fNIRS Signals // Proc of the IEEE 12th Annual Computing and Communication Workshop and Confe-rence. Washington, USA: IEEE, 2022: 1024-1028.
[81] LI Y, ZHANG X, MING D. Early-Stage Fusion of EEG and fNIRS Improves Classification of Motor Imagery. Frontiers in Neuroscien-ce, 2023, 16. DOI: 10.3389/fnins.2022.1062889.
[82] SHELISHIYAH R, THIYAM D B, MARGARET M J, et al. A Hybrid CNN Model for Classification of Motor Tasks Obtained from Hybrid BCI System. Scientific Reports, 2025, 15(1). DOI: 10.1038/s41598-024-84883-2.
[83] MUGHAL N E, KHAN M J, KHALIL K, et al. EEG-fNIRS-Based Hybrid Image Construction and Classification Using CNN-LSTM. Frontiers in Neurorobotics, 2022, 16. DOI: 10.3389/fnbot.2022.873239.
[84] ZHANG G Y, ETEMAD A.Capsule Attention for Multimodal EEG-EOG Representation Learning with Application to Driver Vigilance Estimation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 1138-1149.
[85] COONEY C, FOLLI R, COYLE D.A Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech. IEEE Transactions on Biomedical Engineering, 2022, 69(6): 1983-1994.
[86] GORNALE S S, PALAIAHNAKOTE S, UNKI A, et al. A Novel EEG-ECG Based Fusion Model for Multimodal Emotional Classification. International Journal of Pattern Recognition and Artificial Intelligence, 2026. DOI: 10.1142/S0218001426590081.
[87] ZHANG Y, YIN B, YUAN X Y, et al. Classifier Design for EEG-fNIRS Bimodal Brain-Computer Interface Technology // Proc of the IEEE 20th Conference on Industrial Electronics and Applications. Washington, USA: IEEE, 2025. DOI: 10.1109/ICIEA65512.2025.11149172.
[88] ZHANG Y K, QIU S, HE H G.Multimodal Motor Imagery Decoding Method Based on Temporal Spatial Feature Alignment and Fusion. Journal of Neural Engineering, 2023, 20(2). DOI: 10.1088/1741-2552/acbfdf.
[89] QIU L N, FENG W S, ZHONG L Q, et al. Mutual Information Learning-Based End-to-End Fusion Network for Hybrid EEG-fNIRS Brain-Computer Interface. IEEE Transactions on Instrumentation and Measurement, 2025, 74. DOI: 10.1109/TIM.2025.3604929.
[90] FAISAL M, SAHOO S, HAZARIKA J.STeCANet: Spatio-Temporal Cross Attention Network for Brain Computer Interface Systems Using EEG-fNIRS Signals. Journal of Neural Engineering, 2025, 22(6). DOI: 10.1088/1741-2552/ae2954.
[91] CHENG X Y, WEI W, DU C D, et al. VigilanceNet: Decouple Intra- and Inter-Modality Learning for Multimodal Vigilance Estimation in RSVP-Based BCI // Proc of the 30th ACM International Conference on Multimedia. New York, USA: 2022: 209-217.
[92] SHI X B, WANG H Y, LI B J, et al. Fusion Analysis of EEG-fNIRS Multimodal Brain Signals: A Multitask Classification Algorithm Incorporating Spatial-Temporal Convolution and Dual Attention Mechanisms. IEEE Transactions on Instrumentation and Mea-surement, 2025, 74. DOI: 10.1109/TIM.2025.3538086.
[93] ORTEGA P, ZHAO T, FAISAL A.Deep Real-Time Decoding of Bimanual Grip Force from EEG & fNIRS // Proc of the 10th International IEEE/EMBS Conference on Neural Engineering. Wa-shington, USA: IEEE, 2021: 714-717.
[94] DING P, YIN L Y, ZHOU Z X, et al. MSTFDN: An EEG-fNIRS Multimodal Spatial-Temporal Fusion Decoding Network for Perso-nalized Multi-task Scenarios. Information Fusion, 2026: 131. DOI: 10.1016/j.inffus.2026.104187.
[95] ZHANG L Y, LI B J, CAO M L, et al. Classification of EEG-fNIRS Bimodal Brain Signals for Motor Imagery Tasks Based on Wavelet Transform and Spatio-Temporal Domain Processing. Neuroscience, 2026, 594: 42-47.
[96] HE Z X, CAI M H, LI L T, et al. EEG-EMG FAConformer: Frequency Aware Conv-Transformer for the Fusion of EEG and EMG // Proc of the IEEE International Conference on Bioinformatics and Biomedicine. Washington, USA: IEEE, 2024: 3258-3261.
[97] GUO H B, LI S T, LONG G J, et al. Multimodal MRI-EEG Fusion for Brain-Computer Interface Applications Using a Lightweight CNN and Attention in Offline Parkinson's Disease Diagnosis. Biomedical Signal Processing and Control, 2026, 115. DOI: 10.1016/j.bspc.2025.109432.
[98] XU H Q, SHE Q S, MENG M, et al. EFDFNet: A Multimodal Deep Fusion Network Based on Feature Disentanglement for Attention State Classification. Biomedical Signal Processing and Control, 2025, 109. DOI: 10.1016/j.bspc.2025.108042.
[99] MENG M, HAO D Y, GAO Y Y, et al. Decouple and Rebalance: Towards Effective Joint Learning for EEG-fNIRS Integration. Neurocomputing, 2025, 638. DOI: 10.1016/j.neucom.2025.130128.
[100] HUANG W T, SONG X Y, KUANG D Y.A Hierarchical Diffusion-Convolutional Network with Node-Wise Localization for EEG-NIRS-Based Brain-Computer Interface // Proc of the 12th International Winter Conference on Brain-Computer Interface. Wa-shington, USA: IEEE, 2024. DOI: 10.1109/BCI60775.2024.10480493.
[101] YUAN Z Z, LI Y, ZHANG H Y, et al. Decoding Hybrid EEG-fNIRS Upper Limb Motor Execution with Capsule Dynamic Graph Convolutional Neural Network // Proc of the 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2025. DOI: 10.1109/EMBC58623.2025.11251587.
[102] LI H Y, JI H F, YU J, et al. A Sequential Learning Model with GNN for EEG-EMG-Based Stroke Rehabilitation BCI. Frontiers in Neuroscience, 2023, 17. DOI: 10.3389/fnins.2023.1125230.
[103] ZHAO T, XUE Y L, ZHU J, et al. Heterogeneous Graph Convolutional Neural Networks for EEG-fNIRS Bimodal Emotion Recognition // Proc of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington, USA: IEEE, 2025. DOI: 10.1109/ICASSP49660.2025.10888828.
[104] RAHIMI N, KUMAR C, MCLINDEN J, et al. Topology-Aware Multimodal Fusion for Neural Dynamics Representation Learning and Classification. IEEE Sensors Journal, 2024, 24(13): 21062-21073.
[105] CHEN L, YIN Z, GU X L, et al. Neurophysiological Data Augmentation for EEG-fNIRS Multimodal Features Based on a Denoi-sing Diffusion Probabilistic Model. Computer Methods and Programs in Biomedicine, 2025, 261. DOI: 10.1016/j.cmpb.2025.108594.
[106] CAO Y Y, GAO S W, YU H X, et al. A Motor Imagery Classification Model Based on Hybrid Brain-Computer Interface and Multitask Learning of Electroencephalographic and Electromyographic Deep Features. Frontiers in Physiology, 2024, 15. DOI: 10.3389/fphys.2024.1487809.
[107] QIAN J, WANG D, CHEN J M, et al. Hybrid EEG-fNIRS Decoding for Fine Joint Motor Imagery of Unilateral Upper Limb with Two-Stage Hybrid Training // Proc of the 31st International Conference on Neural Information Processing. Berlin, Germany: Springer, 2025: 313-327.
[108] WANG K N, QIU S, WEI W, et al. A Multimodal Approach to Estimating Vigilance in SSVEP-Based BCI. Expert Systems with Applications, 2023, 225. DOI: 10.1016/j.eswa.2023.120177.
[109] RABBANI M H R, ISLAM S M R. Deep Learning Networks Based Decision Fusion Model of EEG and fNIRS for Classification of Cognitive Tasks. Cognitive Neurodynamics, 2024, 18(4): 1489-1506.
[110] XIA T H, XUE B Q, LIU Y H.MSFL-RGF: Multi-stream Feature Learning and Region-Aware Graph Fusion for EEG-fNIRS Brain Decoding. Signal, Image and Video Processing, 2026, 20(3). DOI: 10.1007/s11760-026-05201-z.
[111] MIAO Y H, WANG H Q, WANG W, et al. DuMoNet: Enhancing Emotion Recognition with a Dual-Modality Structure from EEG and ECG // Proc of the 25th International Conference on Digital Signal Processing. Washington, USA: IEEE, 2025. DOI: 10.1109/DSP65409.2025.11074850.
[112] FU L B, LI H Y, JI H F, et al. EEG-EMG Analysis Method in Hybrid Brain Computer Interface for Hand Rehabilitation Trai-ning. Computing and Informatics, 2023, 42(3): 741-761.
[113] YU B L, CAO L, JIA J, et al. E-FNet: A EEG-fNIRS Dual-Stream Model for Brain-Computer Interfaces. Biomedical Signal Processing and Control, 2025, 100(B). DOI: 10.1016/j.bspc.2024.106943.
[114] KWAK Y, SONG W J, KIM S E.FGANet: fNIRS-Guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30: 329-339.
[115] LI H.DeepSyncNet: Deep Synchronized Fusion Network for EEG-fNIRS Multimodal Brain-Computer Interfaces. Biomedical Signal Processing and Control, 2026, 113(C). DOI: 10.1016/j.bspc.2025.109100.
[116] ZHU Z Y, HAN J P, ZHANG Z Y, et al. Identifying the Nature of Grip Force Signals in EEG & fNIRS with Multi-modal Graph Fusion Network // Proc of the 47th Annual International Confe-rence of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2025. DOI: 10.1109/EMBC58623.2025.11254624.
[117] LIU M T, YANG B H, MENG L, et al. STA-Net: Spatial-Temporal Alignment Network for Hybrid EEG-fNIRS Decoding. Information Fusion, 2025, 119. DOI: 10.1016/j.inffus.2025.103023.
[118] SI X P, ZHANG S, YANG Z B, et al. A Bidirectional Cross-Modal Transformer Representation Learning Model for EEG-fNIRS Multimodal Affective BCI. Expert Systems with Applications, 2025, 266. DOI: 10.1016/j.eswa.2024.126081.
[119] QIN Y X, LI B J, WANG W L, et al. ECA-FusionNet: A Hybrid EEG-fNIRS Signals Network for MI Classification. Journal of Neural Engineering, 2025, 22(1). DOI: 10.1088/1741-2552/adaf58.
[120] YUAN X Y, ZHANG Y, ROLFE P.IIMCNet: Intra- and Inter-Modality Correlation Network for Hybrid EEG-fNIRS Brain-Computer Interface. IEEE Journal of Biomedical and Health Informa-tics, 2026, 30(1): 113-124. |
|
|
|
2026, Vol. 39 
