A Synopsis of Machine and Deep Learning in Medical Physics and Radiology

Abstract

Machine learning (ML) and deep learning (DL) technologies introduced in the fields of medical physics, radiology, and oncology have made great strides in the past few years. A good many applications have proven to be an efficacious automated diagnosis and radiotherapy system. This paper outlines DL's general concepts and principles, key computational methods, and resources, as well as the implementation of automated models in diagnostic radiology and radiation oncology research. In addition, the potential challenges and solutions of DL technology are also discussed.

Keywords

• Deep learning
• Machine learning
• Radiology
• Oncology

References

1. Fouad F. The Fourth Industrial Revolution is the AI Revolution an Academy Prospective. IJISCS. 2019;8(5):155-67. doi: 10.30534/ijiscs/2019/01852019
2. Choy G, Khalilzadeh O, Michalski M, et al. Current Applications and Future Impact of Machine Learning in Radiology. Radiology. 2018;288(2):318-28. doi: 10.1148/radiol.2018171820
3. Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical imaging and radiation therapy. Med Phys. 2019;46(1): e1-e36. doi: 10.1002/mp.13264
4. Avanzo M, Trianni A, Botta F, Talamonti C, Stasi M, Iori M. Artificial Intelligence and the Medical Physicist: Welcome to the Machine. Appl Sci. 2021 Jan;11(4):1691. doi: 10.3390/app11041691
5. Shen C, Nguyen D, Zhou Z, Jiang SB, Dong B, Jia X. An introduction to deep learning in medical physics: advantages, potential, and challenges. Phys Med Biol. 2020;65:05TR1. doi: 10.1088/1361-6560/ab6f51
6. EC AI HLEG. A definition of Artificial Intelligence: main capabilities and scientific disciplines. High-Level Expert Group on Artificial Intelligence (AI HLEG). Brussels. 2019. https://digital-strategy.ec.europa.eu/en/library/definition-artificial-intelligence-main-capabilities-and-scientific-disciplines
7. Alpaydin E. Introduction to machine learning. 4th ed. MIT press. Cambridge Massachusetts. 2020.
8. Baraniuk R, Donoho D, Gavish M. The science of deep learning. Proc Natl Acad Sci. U S A. 2020;117(48):30029-32. doi: 10.1073/pnas.2020596117

9. Sutton RS, Barto AG. Reinforcement learning: An introduction. 2nd ed. Cambridge MA: MIT press. 2018.
10. Zaharchuk G, Gong E, Wintermark M, Rubin D, Langlotz CP. Deep Learning in Neuroradiology. AJNR Am J Neuroradiol. 2018;39(10):1776-84. doi:10.3174/ajnr.A5543
11. Nair V, Hinton GE, editors. Rectified linear units improve restricted boltzmann machines. In Icml. 2010.
12. Lecun Y, Bengio Y. Convolutional networks for images, speech, and time-series. In Arbib MA, editor, The handbook of brain theory and neural networks. MIT Press. 1995.
13. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. InInternational Conference on Medical image computing and computer-assisted intervention. Springer, Cham. 2015 Oct 5;234-41. doi: 10.1007/978-3-319-24574-4_28
14. Milletari, F., Navab, N., Ahmadi, S.A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3DV. IEEE. 2016;565–71. doi: 10.1109/3DV.2016.79
15. Hochreiter S, Schmidhuber J. LSTM can solve hard long time lag problems. NIPS. 1997;473-9. https://dl.acm.org/doi/10.5555/3104482.3104587
16. Chung J, Gulcehre C, Cho K, Bengio YJapa. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS deep learning workshop. 2014; arXiv:1412.3555.
17. Baldi P. Autoencoders, unsupervised learning, and deep architectures. In Proc. ICML Workshop Unsupervised Transf. Learn. 2012;27:37-50. https://dl.acm.org/doi/10.5555/3045796.3045801
18. Salakhutdinov R, Hinton G. Deep boltzmann machines. In Artificial intelligence and statistics. PMLR. 2009 Apr 15;5:448-55. http://proceedings.mlr.press/v5/salakhutdinov09a.html
19. Hinton G. Boltzmann Machines. In: Sammut C., Webb G.I. (eds) Encyclopedia of Machine Learning. Springer, Boston, MA. 2011. doi: 10.1007/978-0-387-30164-8_83
20. Kazeminia S, Baur C, Kuijper A, et al. GANs for medical image analysis. Artificial Intelligence in Medicine. 2020 Aug 9;101938. doi: 10.1016/j.artmed.2020.101938
21. Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine. 2017 Nov 9;34(6):26-38. doi: 10.1109/msp.2017.2743240
22. Yeung M, Sala E, Schönlieb CB, Rundo L. Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation. arXiv preprint arXiv:2102.04525. 2021 Feb 8. https://arxiv.org/abs/2102.04525
23. Haji SH, Abdulazeez AM. COMPARISON OF OPTIMIZATION TECHNIQUES BASED ON GRADIENT DESCENT ALGORITHM: A REVIEW. PalArch's Journal of Archaeology of Egypt/Egyptology. 2021 Feb 18;18(4):2715-43. https://www.archives.palarch.nl/index.php/jae/article/view/6705
24. Abadi M, Barham P, Chen J, et al. Tensorflow: A system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), USENIX Association. 2016;265-83. https://arxiv.org/abs/1605.08695
25. Chollet, F. Keras: Deep learning for humans. GitHub. 2015. Retrieved [2021]. Available from: https://github.com/keras-team/keras
26. Paszke, Adam, Sam Gross, et al. Automatic differentiation in pytorch. In NIPS Workshop. 2017. https://openreview.net/forum?id=BJJsrmfCZ
27. Jia Y, Shelhamer E, Donahue J, et al. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of ACM Multimedia. 2014;675–8. doi: 10.1145/2647868.2654889
28. MATLAB®, version 9.10.0.1613233 (R2021a). The Mathworks, Inc. Natick, MA. 2021. https://www.mathworks.com/
29. Clark K, Vendt B, Smith K, et al. The Cancer Imaging Archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging. 2013 Dec 1;26(6):1045-57. doi: 10.1007/s10278-013-9622-7
30. Tomczak,K., Czerwinska,P. and Wiznerowicz,M. The Cancer Genome Atlas (TCGA): an immeasurable source of knowledge. Contemp Oncol. Pozn. 2015;19(1A):A68–A77. doi: 10.5114/wo.2014.47136
31. Liu X, Song L, Liu S, Zhang Y. A review of deep-learning-based medical image segmentation methods. Sustainability. 2021 Jan;13(3):1224. doi: 10.3390/su13031224
32. Dalmış MU, Litjens G, Holland K, et al. Using deep learning to segment breast and fibroglandular tissue in MRI volumes. Med. Phys. 2017;44(2):533-46. doi: 10.1002/mp.12079
33. Nie D, Wang L, Trullo R, et al. Segmentation of craniomaxillofacial bony structures from MRI with a 3D deep-learning based cascade framework. In: Wang Q., Shi Y., Suk HI., Suzuki K. (eds). MLMI. Lecture Notes in Computer Science. Springer. Cham. 2017;10541:266-73. doi: 10.1007/978-3-319-67389-9_31
34. Cheng R, Roth HR, Lay N, et al. Automatic magnetic resonance prostate segmentation by deep learning with holistically nested networks. J. Med. Imaging 2017; 4:041302. doi: 10.1117/1.jmi.4.4.041302
35. Avanzo M, Wei L, Stancanello J, et al. Machine and deep learning methods for radiomics. Med. Phys. 2020;47(5): e185-e202. doi: 10.1002/mp.13678
36. Gao M, Huang S, Pan X, Liao X, Yang R, Liu J. Machine Learning-Based Radiomics Predicting Tumor Grades and Expression of Multiple Pathologic Biomarkers in Gliomas. Front Oncol. 2020; 10:1676. doi: 10.3389/fonc.2020.01676
37. Castaldo R, Cavaliere C, Soricelli A, Salvatore M, Pecchia L, Franzese M. Radiomic and Genomic Machine Learning Method Performance for Prostate Cancer Diagnosis: Systematic Literature Review. J Med Internet Res. 2021;23(4): e22394. doi: 10.2196/22394
38. Mitra S. Deep Learning with Radiogenomics towards Personalized Management of Gliomas. IEEE Rev Biomed Eng. Epub ahead of print. 2012; PMID: 33900921. doi: 10.1109/RBME.2021.3075500
39. Smedley NF, Aberle DR, Hsu W. Using deep neural networks and interpretability methods to identify gene expression patterns that predict radiomic features and histology in non-small cell lung cancer. J Med Imaging (Bellingham). 2021;8(3):031906. doi: 10.1117/1.jmi.8.3.031906
40. Siar M, Teshnehlab M, editors. Brain Tumor Detection Using Deep Neural Network and Machine Learning Algorithm. ICCKE. 2019;363-8. doi: 10.1109/iccke48569.2019.8964846
41. Zhen SH, Cheng M, Tao YB, Wang YF, Juengpanich S, Jiang ZY, et al. Deep Learning for Accurate Diagnosis of Liver Tumor Based on Magnetic Resonance Imaging and Clinical Data. Front Oncol. 2020; 10:680. doi: 10.3389/fonc.2020.00680
42. Jojoa Acosta MF, Caballero Tovar LY, Garcia-Zapirain MB, Percybrooks WS. Melanoma diagnosis using deep learning techniques on dermatoscopic images. BMC Med Imaging. 2021;21(1):6. doi: 10.1186/s12880-020-00534-8
43. Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI. 2017 May; 3462–71. doi: 10.1109/cvpr.2017.
44. Halder A, Dey D, Sadhu AK. Lung Nodule Detection from Feature Engineering to Deep Learning in Thoracic CT Images: a Comprehensive Review. J Digit Imaging. 2020;33(3):655-77. doi: 10.1007/s10278-020-00320-6
45. Dou Q, Chen H, Yu L, et al. Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks. IEEE Trans Med Imaging. 2016;35(5):1182-95. doi: 10.1109/tmi.2016.2528129
46. Yang W, Chen Y, Liu Y, et al. Cascade of multi-scale convolutional neural networks for bone suppression of chest radiographs in gradient domain. Med. Image Anal. 2017; 35:421-33. doi: 10.1016/j.media.2016.08.004
47. Xiang L, Qiao Y, Nie D, et al. Deep auto-context convolutional neural networks for standard-dose PET image estimation from low-dose PET/MRI. Neurocomputing. 2017; 267:406-16. doi: 10.1016/j.neucom.2017.06.048. doi: 10.1016/j.neucom.2017.06.048
48. Fu J, Yang Y, Singhrao K, et al. Deep learning approaches using 2D and 3D convolutional neural networks for generating male pelvic synthetic computed tomography from magnetic resonance imaging. Med Phys. 2019;46(9):3788-98. doi: 10.1002/mp.13672
49. Liu F, Jang H, Kijowski R, Bradshaw T, McMillan AB. Deep Learning MR Imaging-based Attenuation Correction for PET/MR Imaging. Radiology. 2018;286(2):676-84. doi: 10.1148/radiol.2017170700
50. Leynes AP, Yang J, Wiesinger F, et al. Direct PseudoCT generation for pelvis PET/MRI attenuation correction using deep convolutional neural networks with multi-parametric MRI: zero echo-time and dixon deep pseudoCT (ZeDD-CT). J Nucl Med. 2017;59:852–8. doi: 10.2967/jnumed.117.198051
51. Choi H, Lee DS. Generation of structural MR images from amyloid PET: application to MR-less quantification. J. Nucl. Med. 2018;59(7):1111-7. doi: 10.2967/jnumed.117.199414
52. Ben-Cohen A, Klang E, Raskin S.P, Amitai M.M, Greenspan H. Virtual PET Images from CT Data Using Deep Convolutional Networks: Initial Results. In: Tsaftaris S., Gooya A., Frangi A., Prince J. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2017. Lecture Notes in Computer Science. Springer, Cham. 2017;10557: 49-57. doi: 10.1007/978-3-319-68127-6_6
53. Lee JH, Grant BR, Chung JH, Reiser I, Giger ML. Assessment of diagnostic image quality of computed tomography (CT) images of the lung using deep learning. In: Proc. SPIE Medical Imaging. 2018;10573:105731M. doi: 10.1117/12.2292070
54. Esses SJ, Lu X, Zhao T, et al. Automated image quality evaluation of T2‐weighted liver MRI utilizing deep learning architecture. J. Magn. Reson. Imaging. 2018;47(3):723-8. doi: 10.1002/jmri.25779
55. Schillaci O, Scimeca M, Toschi N, Bonfiglio R, Urbano N, Bonanno E. Combining Diagnostic Imaging and Pathology for Improving Diagnosis and Prognosis of Cancer. Contrast Media & Molecular Imaging. 2019;2019(9429761):1-10. doi: 10.1155/2019/9429761
56. Van Dijk LV, Van den Bosch L, Aljabar P, et al. Improving automatic delineation for head and neck organs at risk by Deep Learning Contouring. Radiother. Oncol. 2020 Jan 1; 142:115-23. doi: 10.1016/j.radonc.2019.09.022
57. Wang M, Zhang Q, Lam S, Cai J, Yang R. A Review on Application of Deep Learning Algorithms in External Beam Radiotherapy Automated Treatment Planning. Front Oncol. 2020;10:580919. doi: 10.3389/fonc.2020.580919
58. Shen C, Nguyen D, Chen, et al. Operating a treatment planning system using a deep-reinforcement learning-based virtual treatment planner for prostate cancer intensity-modulated radiation therapy treatment planning. Med. Phys. 2020;47(6):2329-36. doi: 10.1002/mp.14114
59. Moreau G, Francois-Lavet V, Desbordes P, Macq B. Reinforcement Learning for Radiotherapy Dose Fractioning Automation. Biomedicines. 2021 Feb;9(2):214. doi:10.3390/biomedicines9020214
60. Kiser KJ, Fuller CD, Reed VK. Artificial intelligence in radiation oncology treatment planning: a brief overview. J Med Arti Intell. 2019;2:9. doi: 10.21037/jmai.2019.04.02
61. Zhen X, Chen J, Zhong Z, Hrycushko B, Zhou L, Jiang S, et al. Deep convolutional neural network with transfer learning for rectum toxicity prediction in cervical cancer radiotherapy: a feasibility study. Physics in Medicine & Biology. 2017;62(21):8246. doi: 10.1088/1361-6560/aa8d09
62. Kajikawa T, Kadoya N, Ito K, et al. Automated prediction of dosimetric eligibility of patients with prostate cancer undergoing intensity-modulated radiation therapy using a convolutional neural network. Radiol Phys Technol. 2018;11(3):320-7. doi: 10.1007/s12194-018-0472-3
63. Kalet AM, Luk SM, Phillips MHJMp. Radiation therapy quality assurance tasks and tools: the many roles of machine learning. Med. Phys. 2020;47(5): e168-e77. doi: 10.1002/mp.13445
64. Luo Y, Chen S, Valdes GJMp. Machine learning for radiation outcome modeling and prediction. Med. Phys. 2020;47(5): e178-e84. doi: 10.1002/mp.13570
65. Huynh BQ, Antropova N, Giger ML. Comparison of breast DCE-MRI contrast time points for predicting response to neoadjuvant chemotherapy using deep convolutional neural network features with transfer learning. In: Armato SG, Petrick NA, eds. Proc. SPIE Medical Imaging. 2017;Vol. 10134:101340U. doi: 10.1117/12.2255316
66. Cha KH, Hadjiiski L, Chan H-P, et al. Bladder cancer treatment response assessment in CT using radiomics with deep-learning. Sci. Rep. 2017;7(1):1-12. doi: 10.1038/s41598-017-09315-w
67. Belkin M, Hsu D, Ma S, Mandal S. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS. 2019 Aug 6;116(32):15849-54. doi: 10.1073/pnas.1903070116
68. Affane A, Lebre MA, Mittal U, Vacavant A. Literature Review of Deep Learning Models for Liver Vessels Reconstruction. In: IPTA. IEEE. 2020 Nov 9;1-6. doi: 10.1109/ipta50016.2020.9286639