Deep Learning Based Medical Image Registration: A Review
YING Shihui1, YANG Wan1, DU Shaoyi2, SHI Jun3
1. Department of Mathematics, College of Sciences, Shanghai Uni-versity, Shanghai 200444 2. Institute of Artificial Intelligence and Robotics, College of Artificial Intelligence, Xi′an Jiaotong University, Xi′an 710049 3. School of Communication and Information Engineering, Shang-hai University, Shanghai 200444
Abstract Image registration is a key technology in the field of medical image processing and intelligent analysis. The real-time registration cannot be accomplished due to the high complexity and computational cost of traditional registration methods. With the development of deep learning, learning based image registration methods achieve remarkable results. In this paper, the medical image registration methods based on deep learning are systematically summarized and divided into three categories, including supervised learning, unsupervised learning and dual supervised learning. On this basis, the advantages and disadvantages for each category are discussed. Furthermore, the regularization methods proposed in recent years are emphatically discussed, especially based on diffeomorphism and multi-scale regularization. Finally, the medical image registration methods based on deep learning are prospected according to the development trend of the current medical image registration methods.
Fund:National Natural Science Foundation of China(No.11971296,81627804,61671281), Project of Shanghai Science and Technology Commission of Shanghai Municipality(No.18010 500600,17411953400)
Corresponding Authors:
SHI Jun, Ph.D., professor. His research interests include me-dical image analysis.
About author:: YING Shihui, Ph.D., professor. His research interests include medical image registration. YANG Wan, master student. Her research interests include deep learning for medical image registration. DU Shaoyi, Ph.D., professor. His research interests include medical image registration.
[1] GOU S P, CHEN L L, GU Y, et al. Large-Deformation Image Re-gistration of CT-TEE for Surgical Navigation of Congenital Heart Di-sease. Computational and Mathematical Methods in Medicine, 2018. DOI: 10.1155/2018/4687376. [2] HAN X, HOOGEMAN M S, LEVENDAG P C, et al. Atlas-Based Auto-Segmentation of Head and Neck CT Images // Proc of the International Conference on Medical Image Computing and Compu-ter-Assisted Intervention. Berlin, Germany: Springer, 2008: 434-441. [3] LI R J, JIA X, LEWIS J H, et al. Real-Time Volumetric Image Reconstruction and 3D Tumor Localization Based on a Single X-Ray Projection Image for Lung Cancer Radiotherapy. Medical Physics, 2010, 37(6): 2822-2826. [4] DALCA A V, RAKIC M, GUTTAG J, et al. Learning Conditional Deformable Templates with Convolutional Networks[C/OL]. [2021-01-29]. https://arxiv.org/pdf/1908.02738.pdf. [5] XIA M H, LIU B D. Image Registration by "Super-Curves". IEEE Transactions on Imaging Processing, 2004, 13(5): 720-732. [6] SORZANO C O S, THEVENAZ P, UNSER M. Elastic Registration of Biological Images Using Vector-Spline Regularization. IEEE Tran-sactions on Biomedical Engineering, 2005, 52(4): 652-663. [7] ROHDE G K, ALDROUBI A, DAWANT B W. The Adaptive Bases Algorithm for Intensity-Based Nonrigid Image Registration. IEEE Transactions on Medical Imaging, 2003, 22(11): 1470-1479. [8] BOOKSTEIN F L. Principal Warps: Thin-Plate Splines and the Decomposition of Deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1989, 11(6): 567-585. [9] CHUI H L, RANGARAJAN A. A New Point Matching Algorithm for Non-rigid Registration. Computer Vision and Image Understanding, 2003, 89(2/3): 114-141. [10] JOHNSON H J, CHRISTENSEN G E. Consistent Landmark and Intensity-Based Image Registration. IEEE Transactions on Medical Imaging, 2002, 21(5): 450-461. [11] SHEN D G, DAVATZIKOS C. HAMMER: Hierarchical Attribute Matching Mechanism for Elastic Registration. IEEE Transactions on Medical Imaging, 2002, 21(11): 1421-1439. [12] YING S H, PENG J G, DU S Y, et al. Lie Group Framework of Iterative Closest Point Algorithm for n-D Data Registration. International Journal of Pattern Recognition and Artificial Intelligence, 2009, 23(6): 1201-1220. [13] BEG M F, MILLER M I, TROUVÉ A, et al. Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms. International Journal of Computer Vision, 2005, 61(2): 139-157. [14] VERCAUTEREN T, PENNEC X, PERCHANT A, et al. Diffeomorphic Demons: Efficient Non-parametric Image Registration. NeuroImage, 2009, 45(S1): S61-S72. [15] TROUVÉ A, YOUNES L. Metamorphoses through Lie Group Action. Foundations of Computational Mathematics, 2005, 5(2): 173-198. [16] HOLM D D, TROUVE A, YOUNES L. The Euler-Poincare Theory of Metamorphosis. Quarterly of Applied Mathematics, 2009, 67(4): 661-685. [17] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Cla-ssification with Deep Convolutional Neural Networks // Proc of the 25th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2012: 1097-1105. [18] HASKINS G, KRUGER U, YAN P K. Deep Learning in Medical Image Registration: A Survey. Machine Vision and Applications, 2020, 31(11): 489-497. [19] FU Y B, LEI Y, WANG T H, et al. Deep Learning in Medical Image Registration: A Review[C/OL]. [2020-06-04]. https://arxiv.org/pdf/1912.12318v1.pdf. [20] ZITOVÁ B, FLUSSER J. Image Registration Methods: A Survey. Image and Vision Computing, 2003, 21(11): 977-1000. [21] HILL D L G, BATCHELOR P G, HOLDEN M, et al. Medical Image Registration. Physics in Medicine and Biology, 2001, 46(3): R1-R45. [22] CHEE E, WU Z Z. AIRNet: Self-supervised Affine Registration for 3D Medical Images Using Neural Networks[C/OL]. [2021-01-29]. https://arxiv.org/pdf/1810.02583.pdf. [23] SALEHI S S M, KHAN S, ERDOGMUS D, et al. Real-Time Deep Registration with Geodesic Loss [C/OL]. [2021-01-29]. http://export.arxiv.org/pdf/1803.05982v1. [24] SLOAN J M, GOATMAN K A, SIEBERT J P. Learning Rigid Image Registration-Utilizing Convolutional Neural Networks for Medical Image Registration // Proc of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies. Berlin, Germany: Springer, 2018, II: 89-99. [25] SUN Y Y, MOELKER A, NIESSEN W J, et al. Towards Robust CT-Ultrasound Registration Using Deep Learning Methods // Proc of the International Workshop on Understanding and Interpreting Machine Learning in Medical Image Computing Applications. Berlin, Germany: Springer, 2018: 43-51. [26] EPPENHOF K A I, PLUIM J P W. Pulmonary CT Registration through Supervised Learning with Convolutional Neural Networks. IEEE Transactions on Medical Imaging, 2019, 38(5): 1097-1105. [27] YANG X, KWITT R, NIETHAMMER M. Fast Predictive Image Registration[C/OL]. [2021-01-29]. http://wwwx.cs.unc.edu/~mn/sites/default/files/yang_arxiv_2016.pdf. [28] CAO T, SINGH N, JOJIC V, et al. Semi-coupled Dictionary Learning for Deformation Prediction // Proc of the 12th IEEE International Symposium on Biomedical Imaging(ISBI). Washington, USA: IEEE, 2015: 691-694. [29] ROHÉ M M, DATAR M, HEIMANN T, et al. SVF-Net: Learning Deformable Image Registration Using Shape Matching // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2017: 266-274. [30] CAO X H, YANG J H, ZHANG J, et al. Deformable Image Registration Based on Similarity-Steered CNN Regression // Proc of the International Conference on Medical Image Computing and Compu-ter-Assisted Intervention. Berlin, Germany: Springer, 2017: 300-308. [31] UZUNOVA H, WILMS M, HANDELS H, et al. Training CNNs for Image Registration from Few Samples with Model-Based Data Augmentation // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2017: 223-231. [32] DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet: Learning Optical Flow with Convolutional Networks // Proc of the IEEE International Conference on Computer Vision. Washington, USA: IEEE, 2015: 2758-2766. [33] DE VOS B, BERENDSEN F F, VIERGEVER M A, et al. End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network // Proc of the International Workshop on Deep Learning in Medical Image Analysis. Berlin, Germany: Springer, 2017: 204-212. [34] DE VOS B, BERENDSEN F F, VIERGEVER M A, et al. A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration. Medical Image Analysis, 2019, 52: 128-143. [35] LI H M, FAN Y. Non-rigid Image Registration Using Fully Convolutional Networks with Deep Self-supervision[C/OL]. [2021-01-29]. https://arxiv.org/ftp/arxiv/papers/1709/1709.00799.pdf. [36] GHOSAL S, RAY N. Deep Deformable Registration: Enhancing Accuracy by Fully Convolutional Neural Net. Pattern Recognition Letters, 2017, 94: 81-86. [37] SHEIKHJAFARI A, NOGA M, PUNITHAKUMAR K, et al. Unsupervised Deformable Image Registration with Fully Connected Generative Neural Network[C/OL]. [2021-01-29]. https://openreview.net/pdf?id=HkmkmW2jM. [38] PUNITHAKUMAR K, BOULANGER P, NOGA M. A GPU-Acce-lerated Deformable Image Registration Algorithm with Applications to Right Ventricular Segmentation. IEEE Access, 2017, 5: 20374-20382. [39] STERGIOS C, MIHIR S, MARIA V, et al. Linear and Deformable Image Registration with 3D Convolutional Neural Networks // Proc of the International Workshop on Reconstruction and Analysis for Moving Organ, Breast, and Thoracic Images. Berlin, Germany: Springer, 2018: 13-22. [40] BALAKRISHNAN G, ZHAO A, SABUNCU M R, et al. An Unsupervised Learning Model for Deformable Medical Image Registration // Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2018: 9252-9260. [41] RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutio-nal Networks for Biomedical Image Segmentation // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2015: 234-241. [42] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial Transformer Networks // Proc of the 28th International Conference on Neural Information Processing Systems. Cambridge, USA: The MIT Press, 2015, II: 2017-2025. [43] KUANG D Y, SCHMAH T. FAIM-A ConvNet Method for Unsupervised 3D Medical Image Registration // Proc of the International Workshop on Machine Learning in Medical Imaging. Berlin, Germany: Springer, 2019: 646-654. [44] VIALARD F X, RISSER L, RUECKERT D, et al. Diffeomorphic 3D Image Registration via Geodesic Shooting Using an Efficient Adjoint Calculation. International Journal of Computer Vision, 2012, 97(2): 229-241. [45] SUN L, ZHANG S T. Deformable MRI-Ultrasound Registration Using 3D Convolutional Neural Network // Proc of the Internatio-nal Workshop on Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation. Berlin, Germany: Springer, 2018: 152-158. [46] YOO I, HILDEBRAND D G C, TOBIN W F, et al. ssEMnet: Se-rial-Section Electron Microscopy Image Registration Using a Spatial Transformer Network with Learned Features // Proc of the International Workshop on Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Berlin, Germany: Springer, 2017: 249-257. [47] ARGANDA-CARRERAS I, SORZANO C O S, MARABINI R, et al. Consistent and Elastic Registration of Histological Sections Using Vector-Spline Regularization // Proc of the International Workshop on Computer Vision Approaches to Medical Image Analysis. Berlin, Germany: Springer, 2006: 85-95. [48] FAN J F, CAO X H, XUE Z, et al. Adversarial Similarity Network for Evaluating Image Alignment in Deep Learning Based Re-gistration // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2018: 739-746. [49] LEI Y, FU Y B, HARMS J, et al. 4D-CT Deformable Image Re-gistration Using an Unsupervised Deep Convolutional Neural Network // Proc of the International Workshop on Artificial Intelligence in Radiation Therapy. Berlin, Germany: Springer, 2019: 26-33. [50] FU Y B, LEI Y, WANG T H, et al. LungRegNet: An Unsupervised Deformable Image Registration Method for 4D-CT Lung. Medical Physics, 2020, 47(4): 1763-1774. [51] FAN J F, CAO X H, WANG Q, et al. Adversarial Learning for Mono- or Multi-modal Registration. Medical Image Analysis, 2019. DOI: 10.1016/j.media.2019.101545. [52] FAN J F, CAO X H, YAP P T, et al. BIRNet: Brain Image Re-gistration Using Dual-Supervised Fully Convolutional Networks. Medical Image Analysis, 2019, 54: 193-206. [53] YAN P K, XU S, RASTINEHAD A R, et al. Adversarial Image Registration with Application for MR and TRUS Image Fusion // Proc of the International Workshop on Machine Learning in Medical Imaging. Berlin, Germany: Springer, 2018: 197-204. [54] CAO X H, YANG J H, WANG L, et al. Deep Learning Based Inter-modality Image Registration Supervised by Intra-modality Similarity // Proc of the International Workshop on Machine Learning in Medical Imaging. Berlin, Germany: Springer, 2018: 55-63. [55] HU Y P, MODAT M, GIBSON E, et al. Label Driven Weakly-Supervised Learning for Multimodal Deformable Image Registration // Proc of the 15th IEEE International Symposium on Biomedical Imaging. Washington, USA: IEEE, 2018: 1070-1074. [56] HU Y P, MODAT M, GIBSON E, et al. Weakly-Supervised Con-volutional Neural Networks for Multimodal Image Registration. Medical Image Analysis, 2018, 49: 1-13. [57] ARSIGNY V, COMMOWICK O, PENNEC X, et al. A Log-Euclidean Framework for Statistics on Diffeomorphisms // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2006: 924-931. [58] DALCA A V, BALAKRISHNAN G, GUTTAG J, et al. Unsupervised Learning for Fast Probabilistic Diffeomorphic Registration // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2018: 729-738. [59] KREBS J, MANSI T, MAILHE B, et al. Unsupervised Probabilistic Deformation Modeling for Robust Diffeomorphic Registration // Proc of the International Workshop on Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Su-pport. Berlin, Germany: Springer, 2018: 101-109. [60] ZHANG J. Inverse-Consistent Deep Networks for Unsupervised Deformable Image Registration[C/OL]. [2021-01-29]. https://arxiv.org/pdf/1809.03443.pdf. [61] LU Z W, YANG G Y, HUA T C, et al. Unsupervised Three-Dimensional Image Registration Using a Cycle Convolutional Neural Network // Proc of the IEEE International Conference on Image Processing. Washington, USA: IEEE, 2019: 2174-2178. [62] KIM B, KIM J, LEE J, et al. Unsupervised Deformable Image Registration Using Cycle-Consistent CNN // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2019: 166-174. [63] MOK T C W, CHUNG A C S. Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networks // Proc of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington, USA: IEEE, 2020: 4643-4652. [64] HUANG G, CHEN D L, LI T H, et al. Multi-scale Dense Convolutional Networks for Efficient Prediction[C/OL]. [2021-01-29]. https://arxiv.org/pdf/1703.09844v1.pdf. [65] ZHAO H S, SHI J P, QI X J, et al. Pyramid Scene Parsing Network // Proc of the IEEE Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2017: 6230-6239. [66] DENG L Y, YANG M, LI T Y, et al. RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation[C/OL]. [2021-01-29]. https://arxiv.org/pdf/1907.00135v2.pdf. [67] SOKOOTI H, DE VOS B, BERENDSEN F, et al. Nonrigid Image Registration Using Multi-scale 3D Convolutional Neural Networks // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Sprin-ger, 2017: 232-239. [68] ZHAO S Y, LAU T F, LUO J, et al. Unsupervised 3D End-to-End Medical Image Registration with Volume Tweening Network. IEEE Journal of Biomedical and Health Informatics, 2020, 24(5): 1394-1404. [69] ZHAO S Y, DONG Y, CHANG E, et al. Recursive Cascaded Net-works for Unsupervised Medical Image Registration // Proc of the IEEE/CVF International Conference on Computer Vision. Wa-shington, USA: IEEE, 2019: 10599-10609. [70] JIANG Z R, YIN F F, GE Y, et al. A Multi-scale Framework with Unsupervised Joint Training of Convolutional Neural Networks for Pulmonary Deformable Image Registration. Physics in Medicine and Biology, 2020, 65(1). DOI: 10.1088/1361-6560/ab5da0. [71] WU G R, KIM M, WANG Q, et al. Scalable High-Performance Image Registration Framework by Unsupervised Deep Feature Re-presentations Learning. IEEE Transactions on Biomedical Engineering, 2016, 63(7): 1505-1516. [72] HEINRICH P M. Closing the Gap between Deep and Conventional Image Registration Using Probabilistic Dense Displacement Networks // Proc of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2019: 50-58.
2021, Vol. 34 
