Automatic Generation of Lung Description in Chest X-Ray Based on Deep Learning
HUANG Xin1,2, GU Mengdan1,2, YI Yugen1, CAO Yuanlong1
1. School of Software, Jiangxi Normal University, Nanchang 330022 2. College of Electronic and Information Engineering, Tongji University, Shanghai 201804
Abstract The chest X-ray report automatic generation is a hot research topic in computer-aided diagnosis. More than 65% of diseases in chest X-rays are related to the lungs. For the generation of Chinese reports on lung descriptions, a hierarchical long short term memory model based on semantic labels is proposed. Firstly, the abnormal chest X-ray reports are analyzed, and high-frequency keywords are extracted as semantic labels. Then, the abnormal binary-classification module is introduced to correct the semantic label classification results. Finally, semantic labels and image features are fused to enhance the association mapping between them. Experimental results show that the proposed model is superior to the baseline method in both general and domain metrics, and it improves the performance of chest radiograph report generation effectively.
Fund:National Natural Science Foundation of China(No.61962026), Youth Key Project of Natural Science Foundation of Jiangxi Province(No.20192ACBL21031), Science and Techno-logy Research Project of Jiangxi Provincial Department of Education(No.GJJ200318)
Corresponding Authors:
CAO Yuanlong, Ph.D., associate professor. His research interests include machine learning and network security.
About author:: HUANG Xin, Ph.D., lecturer. His research interests include machine learning, bioinformatics and multi-modal data fusion. GU Mengdan, master student. Her research interests include machine learning and medical information. YI Yugen, Ph.D., associate professor. His research interests include artificial intelligence, computer vision and machine lear-ning.
[1] RAJPURKA P, IRVIN J, ZHU K, et al. ChexNet: Radiologist-Le-vel Pneumonia Detection on Chest X-Rays with Deep Learning[C/OL]. [2021-02-14]. https://arxiv.org/pdf/1711.05225v2.pdf. [2] YAO L, POBLENZ E, DAGUNTS D, et al. Learning to Diagnose from Scratch by Exploiting Dependencies among Labels[C/OL]. [2021-02-14]. https://arxiv.org/pdf/1710.10501v1.pdf. [3] 黄 欣,方 钰,顾梦丹.基于卷积神经网络的X线胸片疾病分类研究.系统仿真学报, 2020, 32(6): 1188-1194. (HUANG X, FANG Y, GU M D. Classification of Chest X-Ray Disease Based on Convolutional Neural Network. Journal of System Simulation, 2020, 32(6): 1188-1194.) [4] DEMNER-FUSHMAN D, KOHLI M D, ROSENMAN M B, et al. Preparing a Collection of Radiology Examinations for Distribution and Retrieval. Journal of the American Medical Informatics Association, 2016, 23(2): 304-310. [5] VINYALS O, TOSHEV A, BENGIO S, et al. Show and Tell: A Neural Image Caption Generator // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2015: 3156-3164. [6] HE X W, YANG Y, SHI B G, et al. VD-SAN: Visual-Densely Semantic Attention Network for Image Caption Generation. Neurocomputing, 2019, 328: 48-55. [7] YAO T, PAN Y W, LI Y H, et al. Boosting Image Captioning with Attributes // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2017: 4904-4912. [8] 张家硕,洪 宇,李志峰,等.基于双向注意力机制的图像描述生成.中文信息学报, 2020, 34(9): 53-61. (ZHANG J S, HONG Y, LI Z F, et al. Image Captioning Based on Bidirectional Attention Mechanism. Journal of Chinese Information Processing, 2020, 34(9): 53-61.) [9] 李志欣,魏海洋,黄飞成,等.结合视觉特征和场景语义的图像描述生成.计算机学报, 2020, 43(9): 1624-1640. (LI Z X, WEI H Y, HUANG F C, et al. Combine Visual Features and Scene Semantics for Image Captioning. Chinese Journal of Computers, 2020, 43(9): 1624-1640.) [10] 毕健旗,刘茂福,胡慧君,等.基于依存句法的图像描述文本生成.北京航空航天大学学报, 2021, 47(3): 431-440. (BI J Q, LIU M F, HU H J, et al. Image Captioning Based on Dependency Syntax. Journal of Beijing University of Aeronautics and Astronautics, 2021, 47(3): 431-440.) [11] SHIN H C, ROBERTS K, LU L, et al. Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2016: 2497-2506. [12] WANG X S, PENG Y F, LU L, et al. TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-Rays // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2018: 9049-9058. [13] JING B Y, XIE P T, XING E. On the Automatic Generation of Medical Imaging Reports // Proc of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2018: 2577-2586. [14] LI C Y, LIANG X D, HU Z T, et al. Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation // Proc of the 32nd International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2018: 1537-1547. [15] XUE Y, XU T, LONG L R, et al. Multimodal Recurrent Model with Attention for Automated Radiology Report Generation // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2018: 457-466. [16] LI C Y, LIANG X D, HU Z T, et al. Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation // Proc of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2019: 6666-6673. [17] HUANG X, YAN F Q, XU W, et al. Multi-attention and Incorporating Background Information Model for Chest X-Ray Image Report Generation. IEEE Access, 2019, 7: 154808-154817. [18] WANG X S, PENG Y F, LU L, et al. Chest X-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 3462-3471. [19] IRVIN J, RAJPURKAR P, KO M, et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison // Proc of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2019: 590-597. [20] JOHNSON A E W, POLLARD T J, BERKOWITZ S J, et al. MIMIC-CXR, a De-identified Publicly Available Database of Chest Radiographs with Free-Text Reports. Scientific Data, 2019, 6(1). DOI: 10.1038/s41597-019-0322-0. [21] YAN F Q, HUANG X, YAO Y, et al. Combining LSTM and DenseNet for Automatic Annotation and Classification of Chest X-Ray Images. IEEE Access, 2019, 7: 74181-74189. [22] HE K M, ZHANG X Y, REN S Q, et al. Deep Residual Learning for Image Recognition // Proc of the IEEE Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2016: 770-778. [23] HOCHREITER S, SCHMIDHUBER J. Long Short-Term Memory. Neural Computation, 1997, 9(8): 1735-1780. [24] XU K, BA J L, KIROS R, et al. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention // Proc of the 32nd International Conference on Machine Learning. New York, USA: ACM, 2015: 2048-2057. [25] LU J S, XIONG C M, PARIKH D, et al. Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning // Proc of the IEEE Conference on Computer Vision and Pattern Re-cognition. Washington, USA: IEEE, 2017: 375-383. [26] CHEN S Z, JIN Q, WANG P, et al. Say as You Wish: Fine-Grained Control of Image Caption Generation with Abstract Scene Graphs // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 9959-9968. [27] PAN Y W, YAO T, LI Y H, et al. X-linear Attention Networks for Image Captioning // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 10968-10977. [28] LIU G X, HSU T M H, MCDERMOTT M, et al. Clinically Accurate Chest X-Ray Report Generation[C/OL]. [2021-02-14]. http://proceedings.mlr.press/v106/liu19a/liu19a.pdf.
2021, Vol. 34 
