Abstract：The accurate identification and classification of pathological images are crucial for early disease detection and treatment. In the process of diagnosis, pathologists typically employ a multi-level approach, observing abnormal cell regions at various magnifications. However, existing models often extract features at a single granularity, neglecting the multi-granularity nature of cells. Therefore, a fuzzy logic guided deep neural network with multi-granularity is proposed in this paper. Firstly, multi-granularity feature extraction is conducted for cell structures at three levels of granularity-coarse, medium and fine, and thus the information in pathological tissue images is fully utilized. Additionally, to address the issue of key information redundancy during multi-granularity feature extraction, fuzzy logic theory is introduced. Multiple fuzzy membership functions are set to describe cell features from different angles. Subsequently, fuzzy operations are employed to fuse into a fuzzy universal feature. A fuzzy logic guided cross attention mechanism is designed to guide the multi-granularity features by the universal fuzzy feature. Finally, an encoder is utilized to propagate the features to all patch tokens, resulting in improved classification accuracy and robustness. Experiments demonstrate that the proposed network achieves high accuracy in the classification of pathological images.
[1] YANG H, CHEN L L, CHENG Z Q, et al. Deep Learning-Based Six-Type Classifier for Lung Cancer and Mimics from Histopathological Whole Slide Images: A Retrospective Study. BMC Medicine, 2021, 19(1). DOI: 10.1186/s12916-021-01953-2. [2] WANG X, CHEN H, GAN C X, et al. Weakly Supervised Deep Learning for Whole Slide Lung Cancer Image Analysis. IEEE Transactions on Cybernetics, 2020, 50(9): 3950-3962. [3] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need // Proc of the 31st International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 6000-6010. [4] WAHAB N, KHAN A, LEE Y S.Transfer Learning Based Deep CNN for Segmentation and Detection of Mitoses in Breast Cancer Histopathological Images. Microscopy, 2019, 68(3): 216-233. [5] SADAFI A, MAKHRO A, BOGDANOVA A, et al. Attention Based Multiple Instance Learning for Classification of Blood Cell Disorders // Proc of the International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2020: 246-256. [6] VALANARASU J M J, OZA P, HACIHALILOGLU I, et al. Medical Transformer: Gated Axial-Attention for Medical Image Segmentation // Proc of the International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 36-46. [7] LI L Q, PAN X P, YANG H H, et al. Multi-task Deep Learning for Fine-Grained Classification and Grading in Breast Cancer Histopa-thological Images. Multimedia Tools and Applications, 2020, 79(21): 14509-14528. [8] HASHIMOTO N, FUKUSHIMA D, KOGA R, et al. Multi-scale Do-main-Adversarial Multiple-Instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 3851-3860. [9] SINHA A, DOLZ J.Multi-scale Self-Guided Attention for Medical Image Segmentation. IEEE Journal of Biomedical and Health Informatics, 2021, 25(1): 121-130. [10] XUE C, ZHU L, FU H Z, et al. Global Guidance Network for Breast Lesion Segmentation in Ultrasound Images. Medical Image Analysis, 2021, 70. DOI: 10.1016/j.media.2021.101989. [11] DING W P, NAYAK J, NAIK B, et al. Fuzzy and Real-Coded Chemical Reaction Optimization for Intrusion Detection in Indus-trial Big Data Environment. IEEE Transactions on Industrial Informatics, 2021, 17(6): 4298-4307. [12] DING W P, CHAKRABORTY S, MALI K, et al. An Unsupervised Fuzzy Clustering Approach for Early Screening of COVID-19 from Radiological Images. IEEE Transactions on Fuzzy Systems, 2022, 30(8): 2902-2914. [13] MURUGESAN G, AHMED T I, BHOLA J, et al. Fuzzy Logic-Based Systems for the Diagnosis of Chronic Kidney Disease. Bio-Med Research International, 2022. DOI: 10.1155/2022/2653665. [14] DING W P, ABDEL-BASSET M, ELDRANDALY K A, et al. Smart Supervision of Cardiomyopathy Based on Fuzzy Harris Hawks Optimizer and Wearable Sensing Data Optimization: A New Mo-del. IEEE Transactions on Cybernetics, 2021, 51(10): 4944-4958. [15] KHAN S, NASEER M, HAYAT M, et al. Transformers in Vision: A Survey. ACM Computing Surveys, 2022, 54(10). DOI: 10.1145/3505244. [16] 汪美玲,邵伟,张道强.基于标签对齐的多模态一致性表型关联方法.软件学报, 2022, 33(12): 4545-4558. (WANG M L, SHAO W, ZHANG D Q.Label-Aligned Multi-modality Consistent Phenotype Association Method. Journal of Software, 2022, 33(12): 4545-4558) [17] 张道强,朱旗,郝小可,等.脑影像智能分析.中国科学(信息科学), 2018, 48(5): 589-602. (ZHANG D Q, ZHU Q, HAO X K, et al. Intelligent Analysis of Brain Images. SCIENTIA SINICA Informationis, 2018, 48(5): 589-602.) [18] WANG W H, XIE E Z, LI X, et al. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 548-558. [19] ZHENG S X, LU J C, ZHAO H S, et al. Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2021: 6877-6886. [20] HAN K, XIAO A, WU E H, et al. Transformer in Transformer[C/OL].[2023-06-06]. https://arxiv.org/abs/2103.0011. [21] DAI Y, GAO Y F, LIU F Y.TransMed: Transformers Advance Multi-modal Medical Image Classification. Diagnostics, 2021, 11(8). DOI: 10.3390/diagnostics11081384. [22] TANG W, HE F Z, LIU Y, et al. MATR: Multimodal Medical Image Fusion via Multiscale Adaptive Transformer. IEEE Transactions on Image Processing, 2022, 31: 5134-5149. [23] WANG S N, LI B Z, KHABSA M, et al. Linformer: Self-Attention with Linear Complexity[C/OL].[2023-06-06]. https://arxiv.org/pdf/2006.04768.pdf. [24] YUAN L, HOU Q B, JIANG Z H, et al. VOLO: Vision Outlooker for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(5): 6575-6586. [25] CHU X X, TIAN Z, WANG Y Q, et al. Twins: Revisiting the Design of Spatial Attention in Vision Transformers[C/OL].[2023-06-06]. https://arxiv.org/pdf/2104.13840v3.pdf. [26] CHEN C F R, FAN Q F, PANDA R. CrossViT: Cross-Attention Multi-scale Vision Transformer for Image Classification // Proc of the IEEE/CVF International Conference on Computer Vision. Washington, USA: IEEE, 2021: 347-356. [27] TOUVRON H, CORD M, DOUZE M, et al. Training Data-Efficient Image Transformers and Distillation through Attention[C/OL].[2023-06-06]. https://arxiv.org/abs/2012.12877. [28] CHEN Z H, SONG Y, CHANG T H, et al. Generating Radiology Reports via Memory-Driven Transformer[C/OL].[2023-06-06]. https://arxiv.org/pdf/2010.16056.pdf. [29] ZADEH L A. Fuzzy Sets. Information and Control, 1965, 8(3): 338-353. [30] WAN M H, CHEN X Y, ZHAN T M, et al. Sparse Fuzzy Two-Dimensional Discriminant Local Preserving Projection(SF2DDLPP) for Robust Image Feature Extraction. Information Sciences, 2021, 563. DOI: 10.1016/j.ins.2021.02.006. [31] BHALLA K, KOUNDAL D, SHARMA B, et al. A Fuzzy Convolutional Neural Network for Enhancing Multi-focus Image Fusion. Journal of Visual Communication and Image Representation, 2022, 84. DOI: 10.1016/j.jvcir.2022.103485. [32] KUMAR M, ALSHEHRI M, ALGHAMDI R, et al. A DE-ANN Inspired Skin Cancer Detection Approach Using Fuzzy c-means Clustering. Mobile Networks and Applications, 2020, 25: 1319-1329. [33] DING W P, WANG H P, HUANG J S, et al. FTransCNN: Fusing Transformer and a CNN Based on Fuzzy Logic for Uncertain Medical Image Segmentation. Information Fusion, 2023. DOI: 10.1016/j.inffus.2023.101880. [34] HU M D, ZHONG Y, XIE S X, et al. Fuzzy System Based Medical Image Processing for Brain Disease Prediction. Frontiers in Neuroscience, 2021, 15. DOI: 10.3389/fnins.2021.714318. [35] WANG L F, ZHANG J, LIU Y, et al. Multimodal Medical Image Fusion Based on Gabor Representation Combination of Multi-CNN and Fuzzy Neural Network. IEEE Access, 2021, 9: 67634-67647. [36] DAS H, NAIK B, BEHERA H S.Medical Disease Analysis Using Neuro-Fuzzy with Feature Extraction Model for Classification. Informatics in Medicine Unlocked, 2020, 18. DOI: 10.1016/j.imu.2019.100288. [37] DING W P, ABDEL-BASSET M, HAWASH H, et al. Multimodal Infant Brain Segmentation by Fuzzy-Informed Deep Learning. IEEE Transactions on Fuzzy Systems, 2022, 30(4): 1088-1101. [38] ORUJOV F, MASKELIŪNAS R, DAMA$\bar{S}$EVIČIUS R, et al. Fuzzy Based Image Edge Detection Algorithm for Blood Vessel Detection in Retinal Images. Applied Soft Computing, 2020, 94. DOI: 10.1016/j.asoc.2020.106452. [39] ZHANG R J, SHEN J, WEI F S, et al. Medical Image Classification Based on Multi-scale Non-negative Sparse Coding. Artificial Intelligence in Medicine, 2017, 83: 44-51. [40] LIN A L, CHEN B Z, XU J Y, et al. DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation. IEEE Transactions on Instrumentation and Measurement, 2022, 71. DOI: 10.1109/TIM.2022.3178991. [41] KONG Y, GENCHEV G Z, WANG X L, et al. Nuclear Segmentation in Histopathological Images Using Two-Stage Stacked U-Nets with Attention Mechanism. Frontiers in Bioengineering and Biotechnology, 2020, 8. DOI: 10.3389/fbioe.2020.573866. [42] 李玉林.病理学. 第8版.北京:人民卫生出版社,2013. (LI Y L.Pathology. 8th Edition. Beijing, China: People's Medical Publishing House, 2013.) [43] ,赫捷.肿瘤学概论.北京:人民卫生出版社, 2013. (WANG G J, HE J.An Introduction to Oncology. Beijing, China: People's Medical Publishing House, 2013.) [44] 宋今丹. 医学细胞生物学.第5版.北京:人民卫生出版社,2013. (SONG J D. Medical Cell Biology.5th Edition. Beijing, China: People's Medical Publishing House, 2013.) [45] KAUL C, MANANDHAR S, PEARS N.FocusNet: An Attention-Based Fully Convolutional Network for Medical Image Segmentation // Proc of the IEEE 16th International Symposium on Biomedical Imaging. Washington, USA: IEEE, 2019: 455-458. [46] ZHU Q, XU B L, HUANG J S, et al. Deep Multi-modal Discriminative and Interpretability Network for Alzheimer's Disease Diagnosis. IEEE Transactions on Medical Imaging, 2023, 42(5): 1472-1483. [47] ZHANG Z J, FU H Z, DAI H, et al. ET-Net: A Generic Edge-Attention Guidance Network for Medical Image Segmentation // Proc of the International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 442-450. [48] BORKOWSKI A A, BUI M M, THOMAS L B, et al. Lung and Colon Cancer Histopathological Image Dataset(LC25000)[C/OL].[2023-06-06]. https://arxiv.org/abs/1912.12142v. [49] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale[C/OL].[2023-06-06]. https://arxiv.org/pdf/2010.11929.pdf. [50] HUO X Z, SUN G, TIAN S W, et al. HiFuse: Hierarchical Multi-scale Feature Fusion Network for Medical Image Classification[C/OL].[2023-06-06]. https://arxiv.org/pdf/2209.10218.pdf. [51] TOLSTIKHIN I O, HOULSBY N, KOLESNIKOV A, et al. MLP-Mixer: An All-MLP Architecture for Vision[C/OL].[2023-06-06]. https://arxiv.org/pdf/2105.01601v1.pdf. [52] MEHMOOD S, GHAZAL T M, KHAN M A, et al. Malignancy Detection in Lung and Colon Histopathology Images Using Transfer Learning with Class Selective Image Processing. IEEE Access, 2022, 10: 25657-25668. [53] MASUD M, SIKDER N, NAHID A A, et al. A Machine Learning Approach to Diagnosing Lung and Colon Cancer Using a Deep Learning-Based Classification Framework. Sensors, 2021, 21(3). DOI: 10.3390/s21030748. [54] HAGHIGHI F, TAHER M R H, GOTWAY M , et al. DiRA: Discriminative, Restorative, and Adversarial Learning for Self-Supervised Medical Image Analysis // Proc of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition. Washington,USA: IEEE, 2022: 20729-20802. [55] WANG X Y, YANG S, ZHANG J, et al. Transformer-Based Unsupervised Contrastive Learning for Histopathological Image Classification. Medical Image Analysis, 2022, 81. DOI: 10.1016/j.media.2022.102559. [56] MORMONT R, GEURTS P, MARÉE R. Multi-task Pre-training of Deep Neural Networks for Digital Pathology. IEEE Journal of Biomedical and Health Informatics, 2020, 25(2): 412-421. [57] SRINIDHI C L, KIM S W, CHEN F D, et al. Self-Supervised Driven Consistency Training for Annotation Efficient Histopathology Image Analysis. Medical Image Analysis, 2022, 75. DOI: 10.1016/j.media.2021.102256. [58] SONG Z Y, DU P H, YAN J P, et al. Nucleus-Aware Self-Supervised Pretraining Using Unpaired Image-to-Image Translation for Histopathology Images. IEEE Transactions on Medical Imaging, 2023. DOI: 10.1109/TMI.2023.3309971. [59] MOHANTY C, MAHAPATRA S, ACHARYA B, et al. Using Deep Learning Architectures for Detection and Classification of Diabetic Retinopathy. Sensors, 2023, 23(12). DOI: 10.3390/s23125726. [60] KASSANI S H, KASSANI P H, KHAZAEINEZHAD R, et al. Diabetic Retinopathy Classification Using a Modified Xception Architecture // Proc of the IEEE International Symposium on Signal Processing and Information Technology. Washington, USA: IEEE, 2019. DOI: 10.1109/ISSPIT47144.2019.9001846. [61] TAN M X, LE Q V.EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[C/OL]. [2023-06-06].https://arxiv.org/pdf/1905.11946.pdf. [62] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet Cla-ssification with Deep Convolutional Neural Networks. Communications of the ACM, 2017, 60(6): 84-90.
2023, Vol. 36 
