Improved Vision Transformer with Lion Optimizer for Lung Diseases Detection

Year 2024, Volume: 16 Issue: 2, 760 - 776, 30.06.2024

Ishak Pacal

https://doi.org/10.29137/umagd.1469472

Cited By: 1

Abstract

Lung infections, such as pneumonia, bronchitis, tuberculosis, and notably COVID-19 caused by the SARS-CoV-2 virus, have caused widespread devastation globally, resulting in a significant loss of life. Timely and precise diagnosis of these respiratory diseases is crucial in controlling their spread and reducing their deadly impact. However, diagnostic errors can occur due to factors like physician workload and the need for a second opinion. To address these challenges, artificial intelligence-based diagnostic systems, utilizing deep learning algorithms, particularly in the radiology field, have been proposed. In this research, we introduced a novel model based on Multi-Axis Image Transformers, which boasts a reduced parameter count, decreased GPU computational load, real-time diagnostic capabilities, and improved accuracy. Furthermore, we conducted a detailed performance comparison of optimization algorithms, including SGD, Adam, and Lion, with higher results indicating that the Lion optimizer notably enhances the diagnostic capabilities of the proposed MaxViT model, especially in detecting lung infections. Our proposed approach underwent rigorous experimentation using the COVID-QU-Ex dataset, recognized as the most current, comprehensive, and balanced dataset for lung infections and COVID-19. Our method achieved diagnostic accuracy of 97.14%, surpassing existing models while maintaining significantly fewer parameters.

Keywords

Detection of Lung infections, COVID-19, MaxViT, Vision transformer, Deep Learning

Supporting Institution

TÜSEB

Project Number

33934

Thanks

This work was supported by the grant provided by TÜSEB under the “2023-C1-YZ” call and Project No: “33934”. We would like to thank TÜSEB for their financial support and scientific contributions.

Akciğer Hastalıklarının Tespiti için Lion Optimizer ile Geliştirilmiş Görüş Transformatörü

Abstract

Pnömoni, bronşit, tüberküloz ve özellikle SARS-CoV-2 virüsü tarafından neden olan COVID-19 gibi akciğer enfeksiyonları, küresel olarak yaygın yıkıma neden oldu ve önemli bir can kaybına yol açtı. Bu solunum yolu hastalıklarının zamanında ve doğru teşhisi, yayılmanın kontrol altına alınması ve ölümcül etkisinin azaltılması açısından hayati öneme sahiptir. Ancak, hekim iş yükü ve ikinci bir görüşe duyulan ihtiyaç gibi faktörler nedeniyle teşhis hataları ortaya çıkabilir. Bu zorlukları ele almak için, özellikle radyoloji alanında derin öğrenme algoritmalarını kullanan yapay zeka tabanlı teşhis sistemleri önerilmiştir. Bu araştırmada, Çok-Eksenli Görüntü Dönüştürücüler temelli yeni bir model sunduk, bu model, azaltılmış parametre sayısı, azaltılmış GPU hesaplama yükü, gerçek zamanlı teşhis yetenekleri ve artan doğruluk gibi özelliklere sahiptir. Ayrıca, SGD, Adam ve Lion dahil olmak üzere optimizasyon algoritmalarının detaylı bir performans karşılaştırmasını yaptık ve etkili sonuçlar, Lion optimizatörünün MaxViT modelinin teşhis yeteneklerini özellikle akciğer enfeksiyonlarını tespitte önemli ölçüde artırdığını göstermektedir. Önerdiğimiz yaklaşım COVID-QU-Ex veri kümesi kullanılarak sıkı bir şekilde deneyime tabi tutuldu ve bu veri kümesi, akciğer enfeksiyonları ve COVID-19 için en güncel, kapsamlı ve dengeli veri kümesi olarak kabul edilmektedir. Yöntemimiz, mevcut modelleri aşarak %97,14'lük bir teşhis doğruluğuna ulaştı ve bunu yaparken belirgin şekilde daha az parametre kullandı.

Keywords

Akciğer enfeksiyonlarının tespiti, COVID-19, MaxViT, Görü transformatörü, Derin Öğrenme

Project Number

33934

References

• Abdul Gafoor, S., Sampathila, N., Madhushankara, M., & Swathi, K. S. (2022). Deep learning model for detection of COVID-19 utilizing the chest X-ray images. Cogent Engineering, 9(1). https://doi.org/10.1080/23311916.2022.2079221
• Ahmad, M., Usama, ·, Bajwa, I., Mehmood, Y., Muhammad, ·, & Anwar, W. (n.d.). Lightweight ResGRU: a deep learning-based prediction of SARS-CoV-2 (COVID-19) and its severity classification using multimodal chest radiography images. Neural Computing and Applications. https://doi.org/10.1007/s00521-023-08200-0
• Ahmed, U., & Lin, J. C. W. (2023). Robust adversarial uncertainty quantification for deep learning fine-tuning. Journal of Supercomputing, 79(10), 11355–11386. https://doi.org/10.1007/s11227-023-05087-5
• Alshmrani, G. M. M., Ni, Q., Jiang, R., Pervaiz, H., & Elshennawy, N. M. (2023). A deep learning architecture for multi-class lung diseases classification using chest X-ray (CXR) images. Alexandria Engineering Journal, 64, 923–935. https://doi.org/10.1016/J.AEJ.2022.10.053
• Aslan, E. (2024). LSTM-ESA HİBRİT MODELİ İLE MR GÖRÜNTÜLERİNDEN BEYİN TÜMÖRÜNÜN SINIFLANDIRILMASI. Adıyaman Üniversitesi Mühendislik Bilimleri Dergisi, 11(22), 63–81. https://doi.org/10.54365/adyumbd.1391157
• Aslan, E., & Özüpak, Y. (2024). Classification of Blood Cells with Convolutional Neural Network Model. Bitlis Eren Üniversitesi Fen Bilimleri Dergisi, 13(1), 314–326. https://doi.org/10.17798/bitlisfen.1401294
• Aslani, S., & Jacob, J. (2023). Utilisation of deep learning for COVID-19 diagnosis. Clinical Radiology, 78(2), 150–157. https://doi.org/10.1016/J.CRAD.2022.11.006
• Ayan, E., Karabulut, B., & Ünver, H. M. (2022). Diagnosis of Pediatric Pneumonia with Ensemble of Deep Convolutional Neural Networks in Chest X-Ray Images. Arabian Journal for Science and Engineering, 47(2), 2123–2139. https://doi.org/10.1007/S13369-021-06127-Z/FIGURES/12
• Bao, H., Dong, L., Piao, S., & Wei, F. (2021). BEiT: BERT Pre-Training of Image Transformers. Mim, 1–18.
• Beyer, L., Izmailov, P., Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I., & Pavetic, F. (2022). FlexiViT: One Model for All Patch Sizes.
• Bhattacharyya, A., Bhaik, D., Kumar, S., Thakur, P., Sharma, R., & Pachori, R. B. (2022). A deep learning based approach for automatic detection of COVID-19 cases using chest X-ray images. Biomedical Signal Processing and Control, 71. https://doi.org/10.1016/j.bspc.2021.103182
• Chen, C. F., Fan, Q., & Panda, R. (2021). CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. Proceedings of the IEEE International Conference on Computer Vision, 347–356. https://doi.org/10.1109/ICCV48922.2021.00041
• Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-Janua, 1800–1807. https://doi.org/10.1109/CVPR.2017.195
• Ciotti, M., Ciccozzi, M., Terrinoni, A., Jiang, W. C., Wang, C. Bin, & Bernardini, S. (2020). The COVID-19 pandemic. Critical Reviews in Clinical Laboratory Sciences, 57(6), 365–388. https://doi.org/10.1080/10408363.2020.1783198
• Cleverley, J., Piper, J., & Jones, M. M. (2020). The role of chest radiography in confirming covid-19 pneumonia. In The BMJ (Vol. 370). BMJ Publishing Group. https://doi.org/10.1136/bmj.m2426
• Constantinou, M., Exarchos, T., Vrahatis, A. G., & Vlamos, P. (2023). COVID-19 Classification on Chest X-ray Images Using Deep Learning Methods. International Journal of Environmental Research and Public Health, 20(3). https://doi.org/10.3390/ijerph20032035
• Cookson, W. O. C. M., Cox, M. J., & Moffatt, M. F. (2018). New opportunities for managing acute and chronic lung infections. Nature Reviews Microbiology, 16(2), 111–120. https://doi.org/10.1038/nrmicro.2017.122
• Deb, S. D., Jha, R. K., Jha, K., & Tripathi, P. S. (2022). A multi model ensemble based deep convolution neural network structure for detection of COVID19. Biomedical Signal Processing and Control, 71. https://doi.org/10.1016/j.bspc.2021.103126
• Devasia, J., Goswami, H., Lakshminarayanan, S., Rajaram, M., & Adithan, S. (123 C.E.). Deep learning classification of active tuberculosis lung zones wise manifestations using chest X-rays: a multi label approach. Scientific Reports |, 13, 887. https://doi.org/10.1038/s41598-023-28079-0
• Dhiman, G., Chang, V., Kant Singh, K., & Shankar, A. (2022). ADOPT: automatic deep learning and optimization-based approach for detection of novel coronavirus COVID-19 disease using X-ray images. Journal of Biomolecular Structure and Dynamics, 40(13), 5836–5847. https://doi.org/10.1080/07391102.2021.1875049
• Dönmez, E. (2024). Hybrid convolutional neural network and multilayer perceptron vision transformer model for wheat species classification task: E-ResMLP+. European Food Research and Technology, 250(5), 1379–1388. https://doi.org/10.1007/S00217-024-04469-0/TABLES/8
• Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem, 770–778. https://doi.org/10.1109/CVPR.2016.90
• Heo, B., Yun, S., Han, D., Chun, S., Choe, J., & Oh, S. J. (2021). Rethinking Spatial Dimensions of Vision Transformers.
• Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.
• Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. (2016). Densely Connected Convolutional Networks. http://arxiv.org/abs/1608.06993
• Ibrokhimov, B., & Kang, J.-Y. (2022). Deep Learning Model for COVID-19-Infected Pneumonia Diagnosis Using Chest Radiography Images. BioMedInformatics, 2(4), 654–670. https://doi.org/10.3390/biomedinformatics2040043
• Işık, G., & Paçal, İ. (2024). Few-shot classification of ultrasound breast cancer images using meta-learning algorithms. Neural Computing and Applications, 1–13. https://doi.org/10.1007/S00521-024-09767-Y/TABLES/7
• Karaman, A., Karaboga, D., Pacal, I., Akay, B., Basturk, A., Nalbantoglu, U., Coskun, S., & Sahin, O. (2022). Hyper-parameter optimization of deep learning architectures using artificial bee colony (ABC) algorithm for high performance real-time automatic colorectal cancer (CRC) polyp detection. Applied Intelligence. https://doi.org/10.1007/s10489-022-04299-1
• Kılıçarslan, S., Diker, A., Közkurt, C., Dönmez, E., Demir, F. B., & Elen, A. (2024). Identification of multiclass tympanic membranes by using deep feature transfer learning and hyperparameter optimization. Measurement, 229, 114488. https://doi.org/10.1016/J.MEASUREMENT.2024.114488
• Kuzinkovas, D., & Clement, S. (2023). The Detection of COVID-19 in Chest X-rays Using Ensemble CNN Techniques. Information, 14(7), 370. https://doi.org/10.3390/info14070370
• Lee, Y., Hwang, J. W., Lee, S., Bae, Y., & Park, J. (2019). An energy and GPU-computation efficient backbone network for real-time object detection. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2019-June, 752–760. https://doi.org/10.1109/CVPRW.2019.00103
• Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8693 LNCS(PART 5), 740–755. https://doi.org/10.1007/978-3-319-10602-1_48
• Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.
• M., E., L., V.-G., I., W. C. K., J., W., & A., Z. (2010). The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88(2), 303–338.
• Mehta, S., & Rastegari, M. (2021). MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer. 3.
• Nafisah, S. I., Muhammad, G., Hossain, M. S., & AlQahtani, S. A. (2023). A Comparative Evaluation between Convolutional Neural Networks and Vision Transformers for COVID-19 Detection. Mathematics, 11(6). https://doi.org/10.3390/math11061489
• Nayak, S. R., Nayak, D. R., Sinha, U., Arora, V., & Pachori, R. B. (2023). An Efficient Deep Learning Method for Detection of COVID-19 Infection Using Chest X-ray Images. Diagnostics, 13(1). https://doi.org/10.3390/diagnostics13010131
• PACAL, İ. (2022). Deep Learning Approaches for Classification of Breast Cancer in Ultrasound (US) Images. Journal of the Institute of Science and Technology, 1917–1927. https://doi.org/10.21597/jist.1183679
• Pacal, I. (2024a). A novel Swin transformer approach utilizing residual multi-layer perceptron for diagnosing brain tumors in MRI images. International Journal of Machine Learning and Cybernetics. https://doi.org/10.1007/s13042-024-02110-w
• Pacal, I. (2024b). Enhancing crop productivity and sustainability through disease identification in maize leaves: Exploiting a large dataset with an advanced vision transformer model. Expert Systems with Applications, 238. https://doi.org/10.1016/j.eswa.2023.122099
• Pacal, I., & Karaboga, D. (2021). A robust real-time deep learning based automatic polyp detection system. Computers in Biology and Medicine, 134. https://doi.org/10.1016/J.COMPBIOMED.2021.104519
• Pacal, I., Karaboga, D., Basturk, A., Akay, B., & Nalbantoglu, U. (2020). A comprehensive review of deep learning in colon cancer. Computers in Biology and Medicine, 126. https://doi.org/10.1016/J.COMPBIOMED.2020.104003
• Platto, S., Xue, T., & Carafoli, E. (2020). COVID19: an announced pandemic. Cell Death and Disease, 11(9). https://doi.org/10.1038/s41419-020-02995-9
• Podder, P., Das, S. R., Mondal, M. R. H., Bharati, S., Maliha, A., Hasan, M. J., & Piltan, F. (2023). LDDNet: A Deep Learning Framework for the Diagnosis of Infectious Lung Diseases. Sensors, 23(1). https://doi.org/10.3390/s23010480
• Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., & Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3), 211–252. https://doi.org/10.1007/s11263-015-0816-y
• Sedik, A., Hammad, M., Abd El-Samie, F. E., Gupta, B. B., & Abd El-Latif, A. A. (2022). Efficient deep learning approach for augmented detection of Coronavirus disease. Neural Computing and Applications, 34(14), 11423–11440. https://doi.org/10.1007/s00521-020-05410-8
• Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 1–14.
• Soomro, T. A., Zheng, L., Afifi, A. J., Ali, A., Yin, M., & Gao, J. (2022). Artificial intelligence (AI) for medical imaging to combat coronavirus disease (COVID-19): a detailed review with direction for future research. Artificial Intelligence Review, 55(2), 1409–1439. https://doi.org/10.1007/s10462-021-09985-z
• Subramanian, N., Elharrouss, O., Al-Maadeed, S., & Chowdhury, M. (2022). A review of deep learning-based detection methods for COVID-19. In Computers in Biology and Medicine (Vol. 143). Elsevier Ltd. https://doi.org/10.1016/j.compbiomed.2022.105233
• Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2014). Going Deeper with Convolutions.
• Tahir, A. M., Chowdhury, M. E. H., Khandakar, A., Rahman, T., Qiblawey, Y., Khurshid, U., Kiranyaz, S., Ibtehaz, N., Rahman, M. S., Al-Maadeed, S., Mahmud, S., Ezeddin, M., Hameed, K., & Hamid, T. (2021). COVID-19 infection localization and severity grading from chest X-ray images. Computers in Biology and Medicine, 139. https://doi.org/10.1016/j.compbiomed.2021.105002
• Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. http://arxiv.org/abs/1905.11946
• Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020). Training data-efficient image transformers & distillation through attention. 1–22.
• Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., & Li, Y. (2022). MaxViT: Multi-axis Vision Transformer. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13684 LNCS, 459–479. https://doi.org/10.1007/978-3-031-20053-3_27
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 2017-Decem(Nips), 5999–6009.
• Velavan, T. P., & Meyer, C. G. (2020). The COVID-19 epidemic. Tropical Medicine and International Health, 25(3), 278–280. https://doi.org/10.1111/tmi.13383
• Xu, W., Xu, Y., Chang, T., & Tu, Z. (2021). Co-Scale Conv-Attentional Image Transformers. Proceedings of the IEEE International Conference on Computer Vision, 9961–9970. https://doi.org/10.1109/ICCV48922.2021.00983
• Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021). Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet.
• Yue, G., Lin, J., An, Z., & Yang, Y. (2022). Loop Residual Attention Network for Automatic Segmentation of COVID-19 Chest X-ray Images. IEEE Access, PP, 1. https://doi.org/10.1109/ACCESS.2022.3227798
• Yuki, K., Fujiogi, M., & Koutsogiannaki, S. (2020). COVID-19 pathophysiology: A review. Clinical Immunology, 215(April). https://doi.org/10.1016/j.clim.2020.108427