Knowledge-Driven Multi-scale Spatial-Temporal Decoding Network for Speech Brain-Computer Interfaces
JIA Zhihong1, WU Dongrui1, ZENG Yuan2
1. School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074; 2. School of Future Technology, Huazhong University of Science and Technology, Wuhan 430074
Abstract：To address the lack of speech-specific modeling mechanisms in general decoding models, a knowledge-driven multi-scale spatio-temporal decoding network for speech brain-computer interfaces(KMST-Former) is proposed. Based on the low-frequency modulation characteristics of speech-related neural activity, parallel multi-scale temporal branches are constructed to capture neural dynamic patterns at different temporal scales. Scale-specific channel attention mechanisms are introduced into each branch to adaptively model spatial coordination. A two-stage spatio-temporal fusion module is then employed to integrate cross-scale and cross-channel information into unified representations. Finally, a Transformer encoder is used to capture long-range temporal dependencies. Experiments demonstrate the superior overall performance of KMST-Former. The effectiveness of multi-scale spatio-temporal modeling guided by speech prior knowledge for enhancing neural feature discrimination is verified.
[1] MRIDHA M F, DAS S C, KABIR M M, et al. Brain-Computer Interface: Advancement and Challenges. Sensors, 2021, 21(17). DOI: 10.3390/s21175746. [2] 伍冬睿. 精准、安全、隐私保护的脑机接口.中国人工智能学会通讯, 2025, 15(3): 29-35. (WU D R. Precise, Safe, and Privacy-Preserving Brain-Computer Interfaces. Chinese Association for Artificial Intelligence, 2025, 15(3): 29-35.) [3] 彭睿旻,江军,匡光涛,等.基于EEG的癫痫自动检测:综述与展望.自动化学报, 2022, 48(2): 335-350. (PENG R M, JIANG J, KUANG G T, et al. EEG-Based Automatic Epilepsy Detection: Review and Outlook. Acta Automatica Sinica, 2022, 48(2): 335-350.) [4] ABIRI R, BORHANI S, SELLERS E W, et al. A Comprehensive Review of EEG-Based Brain-Computer Interface Paradigms. Journal of Neural Engineering, 2019, 16(1). DOI: 10.1088/1741-2552/aaf12e. [5] DICKHAUS T, SANNELLI C, MÜLLER K R, et al. Predicting BCI Performance to Study BCI Illiteracy. BMC Neuroscience, 2009, 10(S1). DOI: 10.1186/1471-2202-10-S1-P84. [6] BRUMBERG J S, NIETO-CASTANON A, KENNEDY P R, et al. Brain-Computer Interfaces for Speech Communication. Speech Communication, 2010, 52(4): 367-379. [7] SILVA A B, LITTLEJOHN K T, LIU J R, et al. The Speech Neuroprosthesis. Nature Reviews Neuroscience, 2024, 25(7): 473-492. [8] ANUMANCHIPALLI G K, CHARTIER J, CHANG E F.Speech Synthesis from Neural Decoding of Spoken Sentences. Nature, 2019, 568(7753): 493-498. [9] GU X T, CAO Z H, JOLFAEI A,et al. EEG-Based Brain-Compu-ter Interfaces(BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021, 18(5): 1645-1666. [10] DASALLA C S, KAMBARA H, SATO M, et al. Single-Trial Cla-ssification of Vowel Speech Imagery Using Common Spatial Pa-tterns. Neural Networks, 2009, 22(9): 1334-1339. [11] ZHANG L Y, ZHOU Y Y, GONG P L, et al. Speech Imagery Decoding Using EEG Signals and Deep Learning: A Survey. IEEE Transactions on Cognitive and Developmental Systems, 2025, 17(1): 22-39. [12] LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: A Compact Convolutional Neural Network for EEG-Based Brain-Computer Interfaces. Journal of Neural Engineering, 2018, 15(5). DOI: 10.1088/1741-2552/aace8c. [13] SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization. Human Brain Mapping, 2017, 38(11): 5391-5420. [14] SONG Y H, ZHENG Q Q, LIU B C, et al. EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 710-719. [15] MANE R, ROBINSON N, VINOD A P, et al. A Multi-view CNN with Novel Variance Layer for Motor Imagery Brain Computer Interface // Proc of the 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2020: 2950-2953. [16] WANG J H, YAO L, WANG Y M.IFNet: An Interactive Frequency Convolutional Neural Network for Enhancing Motor Imagery Decoding from EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 1900-1911. [17] THUWAJIT P, RANGPONG P, SAWANGJAI P,et al. EEGWave-Net: Multiscale CNN-Based Spatiotemporal Feature Extraction for EEG Seizure Detection. IEEE Transactions on Industrial Informa-tics, 2022, 18(8): 5547-5557. [18] TAO W, WANG Z, WONG C M, et al. ADFCNN: Attention-Based Dual-Scale Fusion Convolutional Neural Network for Motor Imagery Brain-Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024, 32: 154-165. [19] ZHAO W, ZHANG B C, ZHOU H F, et al. Multi-scale Convolutional Transformer Network for Motor Imagery Brain-Computer Interface. Scientific Reports, 2025, 15(1). DOI: 10.1038/s41598-025-96611-5. [20] ALTUWAIJRI G A, MUHAMMAD G, ALTAHERI H, et al. A Multi-branch Convolutional Neural Network with Squeeze-and-Excitation Attention Blocks for EEG-Based Motor Imagery Signals Classification. Diagnostics, 2022, 12(4). DOI: 10.3390/diagnostics12040995. [21] TONG L N, QIAN Y H, PENG L, et al. A Learnable EEG Cha-nnel Selection Method for MI-BCI Using Efficient Channel Attention. Frontiers in Neuroscience, 2023, 17. DOI: 10.3389/fnins.2023.1276067. [22] LIAO W Z, MIAO Z P, LIANG S B, et al. A Composite Improved Attention Convolutional Network for Motor Imagery EEG Classification. Frontiers in Neuroscience, 2025, 19. DOI: 10.3389/fnins.2025.1543508. [23] JIANG L, SIRIARAYA P, CHOI D, et al. Electroencephalogram Signals Emotion Recognition Based on Convolutional Neural Network-Recurrent Neural Network Framework with Channel-Temporal Attention Mechanism for Older Adults. Frontiers in Aging Neuroscience, 2022, 14. DOI: 10.3389/fnagi.2022.945024. [24] DING Y, LI Y, SUN H, et al. EEG-Deformer: A Dense Convolutio-nal Transformer for Brain-Computer Interfaces. IEEE Journal of Biomedical and Health Informatics, 2025, 29(3): 1909-1918. [25] ZHAO W, JIANG X L, ZHANG B C, et al. CTNet: A Convolutional Transformer Network for EEG-Based Motor Imagery Classification. Scientific Reports, 2024, 14(1). DOI: 10.1038/s41598-024-71118-7. [26] LIU K, YANG T, YU Z L, et al. MSVTNet: Multi-scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding. IEEE Journal of Biomedical and Health Informatics, 2024, 28(12): 7126-7137. [27] ZHAO Q, ZHU W N.TMSA-Net: A Novel Attention Mechanism for Improved Motor Imagery EEG Signal Processing. Biomedical Signal Processing and Control, 2025, 102. DOI: 10.1016/j.bspc.2024.107189. [28] WANG Z W, WANG H B, JIA T W, et al. DBConformer: Dual-Branch Convolutional Transformer for EEG Decoding. IEEE Journal of Biomedical and Health Informatics, 2026, 30(5): 4134-4147. [29] JIANG M Y, ZHANG W, DING Y, et al. Decoding Covert Speech from EEG by Functional Areas Spatio-Temporal Transformer. IEEE Journal of Biomedical and Health Informatics, 2026. DOI: 10.1109/JBHI.2026.3653025. [30] STONE M A, FÜLLGRABE C, MOORE B C J . Relative Contribution to Speech Intelligibility of Different Envelope Modulation Rates within the Speech Dynamic Range. The Journal of the Acoustical Society of America, 2010, 128(4): 2127-2137. [31] KEITEL A, GROSS J, KAYSER C.Perceptually Relevant Speech Tracking in Auditory and Motor Cortex Reflects Distinct Linguistic Features. PLoS Biology, 2018, 16(3). DOI: 10.1371/journal.pbio.2004473. [32] LEONG V, GOSWAMI U.Acoustic-Emergent Phonology in the Am-plitude Envelope of Child-Directed Speech. PLoS One, 2015, 10(12). DOI: 10.1371/journal.pone.0144411.
2026, Vol. 39 
