基于深度学习的X线胸片肺部描述自动生成

doi:10.16451/j.cnki.issn1003-6059.202106007

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (806 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要 X线胸片报告的自动生成是计算机辅助诊断研究的热点,X线胸片中65%以上的疾病与肺部相关.针对肺部描述中文报告生成,提出基于语义标签的层级长短期记忆网络模型.首先,分析异常胸片报告,提取高频关键词作为图像语义标签.再加入异常二分类模块,用于修正语义标签分类结果.最后,融合语义标签与图像特征,加强二者的关联映射.实验表明,文中模型在通用和领域指标的评价上均较优,能有效提高胸片报告生成的性能.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	黄欣
	顾梦丹
	易玉根
	曹远龙

关键词 ： X线胸片, 语义标签, 层级长短期记忆网络, 中文报告, 肺部描述

Abstract：The chest X-ray report automatic generation is a hot research topic in computer-aided diagnosis. More than 65% of diseases in chest X-rays are related to the lungs. For the generation of Chinese reports on lung descriptions, a hierarchical long short term memory model based on semantic labels is proposed. Firstly, the abnormal chest X-ray reports are analyzed, and high-frequency keywords are extracted as semantic labels. Then, the abnormal binary-classification module is introduced to correct the semantic label classification results. Finally, semantic labels and image features are fused to enhance the association mapping between them. Experimental results show that the proposed model is superior to the baseline method in both general and domain metrics, and it improves the performance of chest radiograph report generation effectively.

Key words： Chest X-Ray Semantic Label Hierarchical Long Short-Term Memory Chinese Report Lung Description

收稿日期: 2021-03-08

ZTFLH:

TP 391

基金资助:国家自然科学基金项目(No.61962026)、江西省自然科学基金青年重点项目(No.20192ACBL21031)、江西省教育厅科技研究项目(No.GJJ200318)资助

通讯作者: 曹远龙,博士,副教授,主要研究方向为机器学习、网络安全.E-mail:ylcao@jxnu.edu.cn.

作者简介: 黄欣,博士,讲师,主要研究方向为机器学习、生物信息、多模态数据融合.E-mail:xinhuang@jxnu.edu.cn.
顾梦丹,硕士研究生,主要研究方向为机器学习、医学信息.E-mail:1732953@tongji.edu.cn.
易玉根,博士,副教授,主要研究方向为人工智能、计算机视觉、机器学习.E-mail:yiyg510@jxnu.edu.cn.

引用本文:

黄欣, 顾梦丹, 易玉根, 曹远龙. 基于深度学习的X线胸片肺部描述自动生成[J]. 模式识别与人工智能, 2021, 34(6): 552-560. HUANG Xin, GU Mengdan, YI Yugen, CAO Yuanlong. Automatic Generation of Lung Description in Chest X-Ray Based on Deep Learning. , 2021, 34(6): 552-560.

链接本文:

http://manu46.magtech.com.cn/Jweb_prai/CN/10.16451/j.cnki.issn1003-6059.202106007 或 http://manu46.magtech.com.cn/Jweb_prai/CN/Y2021/V34/I6/552

[1] RAJPURKA P, IRVIN J, ZHU K, et al. ChexNet: Radiologist-Le-vel Pneumonia Detection on Chest X-Rays with Deep Learning[C/OL]. [2021-02-14]. https://arxiv.org/pdf/1711.05225v2.pdf.
[2] YAO L, POBLENZ E, DAGUNTS D, et al. Learning to Diagnose from Scratch by Exploiting Dependencies among Labels[C/OL]. [2021-02-14]. https://arxiv.org/pdf/1710.10501v1.pdf.
[3] 黄欣,方钰,顾梦丹.基于卷积神经网络的X线胸片疾病分类研究.系统仿真学报, 2020, 32(6): 1188-1194.
(HUANG X, FANG Y, GU M D. Classification of Chest X-Ray Disease Based on Convolutional Neural Network. Journal of System Simulation, 2020, 32(6): 1188-1194.)
[4] DEMNER-FUSHMAN D, KOHLI M D, ROSENMAN M B, et al. Preparing a Collection of Radiology Examinations for Distribution and Retrieval. Journal of the American Medical Informatics Association, 2016, 23(2): 304-310.
[5] VINYALS O, TOSHEV A, BENGIO S, et al. Show and Tell: A Neural Image Caption Generator // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 3156-3164.
[6] HE X W, YANG Y, SHI B G, et al. VD-SAN: Visual-Densely Semantic Attention Network for Image Caption Generation. Neurocomputing, 2019, 328: 48-55.
[7] YAO T, PAN Y W, LI Y H, et al. Boosting Image Captioning with Attributes // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4904-4912.
[8] 张家硕,洪宇,李志峰,等.基于双向注意力机制的图像描述生成.中文信息学报, 2020, 34(9): 53-61.
(ZHANG J S, HONG Y, LI Z F, et al. Image Captioning Based on Bidirectional Attention Mechanism. Journal of Chinese Information Processing, 2020, 34(9): 53-61.)
[9] 李志欣,魏海洋,黄飞成,等.结合视觉特征和场景语义的图像描述生成.计算机学报, 2020, 43(9): 1624-1640.
(LI Z X, WEI H Y, HUANG F C, et al. Combine Visual Features and Scene Semantics for Image Captioning. Chinese Journal of Computers, 2020, 43(9): 1624-1640.)
[10] 毕健旗,刘茂福,胡慧君,等.基于依存句法的图像描述文本生成.北京航空航天大学学报, 2021, 47(3): 431-440.
(BI J Q, LIU M F, HU H J, et al. Image Captioning Based on Dependency Syntax. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 431-440.)
[11] SHIN H C, ROBERTS K, LU L, et al. Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2016: 2497-2506.
[12] WANG X S, PENG Y F, LU L, et al. TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 9049-9058.
[13] JING B Y, XIE P T, XING E. On the Automatic Generation of Medical Imaging Reports // Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2018: 2577-2586.
[14] LI C Y, LIANG X D, HU Z T, et al. Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation // Proc of the 32nd International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2018: 1537-1547.
[15] XUE Y, XU T, LONG L R, et al. Multimodal Recurrent Model with Attention for Automated Radiology Report Generation // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2018: 457-466.
[16] LI C Y, LIANG X D, HU Z T, et al. Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation // Proc of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2019: 6666-6673.
[17] HUANG X, YAN F Q, XU W, et al. Multi-attention and Incorporating Background Information Model for Chest X-Ray Image Report Generation. IEEE Access, 2019, 7: 154808-154817.
[18] WANG X S, PENG Y F, LU L, et al. Chest X-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 3462-3471.
[19] IRVIN J, RAJPURKAR P, KO M, et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison // Proc of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2019: 590-597.
[20] JOHNSON A E W, POLLARD T J, BERKOWITZ S J, et al. MIMIC-CXR, a De-identified Publicly Available Database of Chest Radiographs with Free-Text Reports. Scientific Data, 2019, 6(1). DOI: 10.1038/s41597-019-0322-0.
[21] YAN F Q, HUANG X, YAO Y, et al. Combining LSTM and DenseNet for Automatic Annotation and Classification of Chest X-Ray Images. IEEE Access, 2019, 7: 74181-74189.
[22] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778.
[23] HOCHREITER S, SCHMIDHUBER J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780.
[24] XU K, BA J L, KIROS R, et al. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention // Proc of the 32nd International Conference on Machine Learning. New York, USA: ACM, 2015: 2048-2057.
[25] LU J S, XIONG C M, PARIKH D, et al. Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 375-383.
[26] CHEN S Z, JIN Q, WANG P, et al. Say as You Wish: Fine-Grained Control of Image Caption Generation with Abstract Scene Graphs // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 9959-9968.
[27] PAN Y W, YAO T, LI Y H, et al. X-linear Attention Networks for Image Captioning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 10968-10977.
[28] LIU G X, HSU T M H, MCDERMOTT M, et al. Clinically Accurate Chest X-Ray Report Generation[C/OL]. [2021-02-14]. http://proceedings.mlr.press/v106/liu19a/liu19a.pdf.