知识驱动的多尺度时空言语脑机接口解码网络

doi:10.16451/j.cnki.issn1003-6059.202606002

Abstract
Figure/Table
References
Related Citation (15)

Download: PDF (1337 KB) HTML (1 KB)
Export: BibTeX | EndNote (RIS)

Abstract To address the lack of speech-specific modeling mechanisms in general decoding models, a knowledge-driven multi-scale spatio-temporal decoding network for speech brain-computer interfaces(KMST-Former) is proposed. Based on the low-frequency modulation characteristics of speech-related neural activity, parallel multi-scale temporal branches are constructed to capture neural dynamic patterns at different temporal scales. Scale-specific channel attention mechanisms are introduced into each branch to adaptively model spatial coordination. A two-stage spatio-temporal fusion module is then employed to integrate cross-scale and cross-channel information into unified representations. Finally, a Transformer encoder is used to capture long-range temporal dependencies. Experiments demonstrate the superior overall performance of KMST-Former. The effectiveness of multi-scale spatio-temporal modeling guided by speech prior knowledge for enhancing neural feature discrimination is verified.

Key words： Brain-Computer Interface Speech Decoding Knowledge-Driven Multi-scale Convolution Attention Mechanism

Received: 06 May 2026

ZTFLH:

TP391.4

Fund:Young Scientists Fund of National Natural Science Foundation of China(No.62525305), Project of Wuhan Science and Technology Bureau(No.2024060702030150), Taihu Lake Innovation Fund for Future Technology(No.HUST 2024-A-1)

Corresponding Authors: WU Dongrui, Ph.D., professor. His research interests include brain-computer interface, machine learning and affec-tive computing.

About author:: About Author: JIA Zhihong, Master student. Her research interests include speech brain-compu-ter interface. ZENG Yuan, Undergraduate student.Academic Papers of the 28th Annual Meeting of the China Association for Science and Technology

	Service

	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	JIA Zhihong
	WU Dongrui
	ZENG Yuan

Cite this article:

JIA Zhihong,WU Dongrui,ZENG Yuan. Knowledge-Driven Multi-scale Spatial-Temporal Decoding Network for Speech Brain-Computer Interfaces[J]. Pattern Recognition and Artificial Intelligence, 2026, 39(6): 489-503.

URL:

http://manu46.magtech.com.cn/Jweb_prai/EN/10.16451/j.cnki.issn1003-6059.202606002 OR http://manu46.magtech.com.cn/Jweb_prai/EN/Y2026/V39/I6/489

[1] MRIDHA M F, DAS S C, KABIR M M, et al. Brain-Computer Interface: Advancement and Challenges. Sensors, 2021, 21(17). DOI: 10.3390/s21175746.
[2] 伍冬睿. 精准、安全、隐私保护的脑机接口.中国人工智能学会通讯, 2025, 15(3): 29-35.
(WU D R. Precise, Safe, and Privacy-Preserving Brain-Computer Interfaces. Chinese Association for Artificial Intelligence, 2025, 15(3): 29-35.)
[3] 彭睿旻,江军,匡光涛,等.基于EEG的癫痫自动检测:综述与展望.自动化学报, 2022, 48(2): 335-350.
(PENG R M, JIANG J, KUANG G T, et al. EEG-Based Automatic Epilepsy Detection: Review and Outlook. Acta Automatica Sinica, 2022, 48(2): 335-350.)
[4] ABIRI R, BORHANI S, SELLERS E W, et al. A Comprehensive Review of EEG-Based Brain-Computer Interface Paradigms. Journal of Neural Engineering, 2019, 16(1). DOI: 10.1088/1741-2552/aaf12e.
[5] DICKHAUS T, SANNELLI C, MÜLLER K R, et al. Predicting BCI Performance to Study BCI Illiteracy. BMC Neuroscience, 2009, 10(S1). DOI: 10.1186/1471-2202-10-S1-P84.
[6] BRUMBERG J S, NIETO-CASTANON A, KENNEDY P R, et al. Brain-Computer Interfaces for Speech Communication. Speech Communication, 2010, 52(4): 367-379.
[7] SILVA A B, LITTLEJOHN K T, LIU J R, et al. The Speech Neuroprosthesis. Nature Reviews Neuroscience, 2024, 25(7): 473-492.
[8] ANUMANCHIPALLI G K, CHARTIER J, CHANG E F.Speech Synthesis from Neural Decoding of Spoken Sentences. Nature, 2019, 568(7753): 493-498.
[9] GU X T, CAO Z H, JOLFAEI A,et al. EEG-Based Brain-Compu-ter Interfaces(BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021, 18(5): 1645-1666.
[10] DASALLA C S, KAMBARA H, SATO M, et al. Single-Trial Cla-ssification of Vowel Speech Imagery Using Common Spatial Pa-tterns. Neural Networks, 2009, 22(9): 1334-1339.
[11] ZHANG L Y, ZHOU Y Y, GONG P L, et al. Speech Imagery Decoding Using EEG Signals and Deep Learning: A Survey. IEEE Transactions on Cognitive and Developmental Systems, 2025, 17(1): 22-39.
[12] LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: A Compact Convolutional Neural Network for EEG-Based Brain-Computer Interfaces. Journal of Neural Engineering, 2018, 15(5). DOI: 10.1088/1741-2552/aace8c.
[13] SCHIRRMEISTER R T, SPRINGENBERG J T, FIEDERER L D J, et al. Deep Learning with Convolutional Neural Networks for EEG Decoding and Visualization. Human Brain Mapping, 2017, 38(11): 5391-5420.
[14] SONG Y H, ZHENG Q Q, LIU B C, et al. EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 710-719.
[15] MANE R, ROBINSON N, VINOD A P, et al. A Multi-view CNN with Novel Variance Layer for Motor Imagery Brain Computer Interface // Proc of the 42nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Washington, USA: IEEE, 2020: 2950-2953.
[16] WANG J H, YAO L, WANG Y M.IFNet: An Interactive Frequency Convolutional Neural Network for Enhancing Motor Imagery Decoding from EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2023, 31: 1900-1911.
[17] THUWAJIT P, RANGPONG P, SAWANGJAI P,et al. EEGWave-Net: Multiscale CNN-Based Spatiotemporal Feature Extraction for EEG Seizure Detection. IEEE Transactions on Industrial Informa-tics, 2022, 18(8): 5547-5557.
[18] TAO W, WANG Z, WONG C M, et al. ADFCNN: Attention-Based Dual-Scale Fusion Convolutional Neural Network for Motor Imagery Brain-Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024, 32: 154-165.
[19] ZHAO W, ZHANG B C, ZHOU H F, et al. Multi-scale Convolutional Transformer Network for Motor Imagery Brain-Computer Interface. Scientific Reports, 2025, 15(1). DOI: 10.1038/s41598-025-96611-5.
[20] ALTUWAIJRI G A, MUHAMMAD G, ALTAHERI H, et al. A Multi-branch Convolutional Neural Network with Squeeze-and-Excitation Attention Blocks for EEG-Based Motor Imagery Signals Classification. Diagnostics, 2022, 12(4). DOI: 10.3390/diagnostics12040995.
[21] TONG L N, QIAN Y H, PENG L, et al. A Learnable EEG Cha-nnel Selection Method for MI-BCI Using Efficient Channel Attention. Frontiers in Neuroscience, 2023, 17. DOI: 10.3389/fnins.2023.1276067.
[22] LIAO W Z, MIAO Z P, LIANG S B, et al. A Composite Improved Attention Convolutional Network for Motor Imagery EEG Classification. Frontiers in Neuroscience, 2025, 19. DOI: 10.3389/fnins.2025.1543508.
[23] JIANG L, SIRIARAYA P, CHOI D, et al. Electroencephalogram Signals Emotion Recognition Based on Convolutional Neural Network-Recurrent Neural Network Framework with Channel-Temporal Attention Mechanism for Older Adults. Frontiers in Aging Neuroscience, 2022, 14. DOI: 10.3389/fnagi.2022.945024.
[24] DING Y, LI Y, SUN H, et al. EEG-Deformer: A Dense Convolutio-nal Transformer for Brain-Computer Interfaces. IEEE Journal of Biomedical and Health Informatics, 2025, 29(3): 1909-1918.
[25] ZHAO W, JIANG X L, ZHANG B C, et al. CTNet: A Convolutional Transformer Network for EEG-Based Motor Imagery Classification. Scientific Reports, 2024, 14(1). DOI: 10.1038/s41598-024-71118-7.
[26] LIU K, YANG T, YU Z L, et al. MSVTNet: Multi-scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding. IEEE Journal of Biomedical and Health Informatics, 2024, 28(12): 7126-7137.
[27] ZHAO Q, ZHU W N.TMSA-Net: A Novel Attention Mechanism for Improved Motor Imagery EEG Signal Processing. Biomedical Signal Processing and Control, 2025, 102. DOI: 10.1016/j.bspc.2024.107189.
[28] WANG Z W, WANG H B, JIA T W, et al. DBConformer: Dual-Branch Convolutional Transformer for EEG Decoding. IEEE Journal of Biomedical and Health Informatics, 2026, 30(5): 4134-4147.
[29] JIANG M Y, ZHANG W, DING Y, et al. Decoding Covert Speech from EEG by Functional Areas Spatio-Temporal Transformer. IEEE Journal of Biomedical and Health Informatics, 2026. DOI: 10.1109/JBHI.2026.3653025.
[30] STONE M A, FÜLLGRABE C, MOORE B C J . Relative Contribution to Speech Intelligibility of Different Envelope Modulation Rates within the Speech Dynamic Range. The Journal of the Acoustical Society of America, 2010, 128(4): 2127-2137.
[31] KEITEL A, GROSS J, KAYSER C.Perceptually Relevant Speech Tracking in Auditory and Motor Cortex Reflects Distinct Linguistic Features. PLoS Biology, 2018, 16(3). DOI: 10.1371/journal.pbio.2004473.
[32] LEONG V, GOSWAMI U.Acoustic-Emergent Phonology in the Am-plitude Envelope of Child-Directed Speech. PLoS One, 2015, 10(12). DOI: 10.1371/journal.pone.0144411.