Hybrid Deep Learning and Reinforcement Learning Approach for Brain Tumor Classification from MRI Images

Year 2025, Volume: 20 Issue: 2, 206 - 221, 26.11.2025

Çiğdem Saraç , Seda Arıkan Arıbal , Yiğit Ali Üncü

Abstract

Brain cancer, resulting from abnormal tumor growth in brain tissue, requires accurate and timely diagnosis. Although MRI plays a crucial role, manual interpretation is prone to errors and delays. To address this, we propose a hybrid system combining deep learning (VGG16, ResNet50, DenseNet201) with reinforcement learning (Q-learning) for brain tumor classification. Using three distinct MRI datasets within MATLAB, the models achieved high classification accuracies: 97.45% (VGG16), 96.06% (ResNet50), and 96.93% (DenseNet201). The integration of reinforcement learning improved decision-making and interpretability. Additionally, a user-friendly interface was developed to support clinical decision-making. This study demonstrates that combining deep learning with reinforcement learning enhances model adaptability, offering a more reliable and effective diagnostic approach.

Keywords

Brain tumor , MRI classification , Deep learning , Reinforcement learning , CNN models

References

• S. Rasheed, K. Rehman and M. S. H. Akash, "An insight into the risk factors of brain tumors and their therapeutic interventions", Biomedicine & Pharmacotherapy, 143, 112119, 2021.
• Q. T. Ostrom, M. Price, C. Neff, G. Cioffi, K. A. Waite, C. Kruchko and J. S. Barnholtz-Sloan, "CBTRUS Statistical Report: Primary brain and other central nervous system tumors diagnosed in the United States in 2015-2019", Neuro-Oncology, 24(Suppl 5), v1-v95, 2022.
• A. M. Molinaro, J. W. Taylor, J. K. Wiencke and M. R. Wrensch, "Genetic and molecular epidemiology of adult diffuse glioma", Nature Reviews Neurology, 15(7), 405-417, 2019.
• A. Thakur, C. Faujdar, R. Sharma, S. Sharma, B. Malik, K. Nepali and J. P. Liou, "Glioblastoma: Current status, emerging targets, and recent advances", Journal of Medicinal Chemistry, 65(13), 8596-8685, 2022.
• T. Wang, Y. Ni and L. Liu, "Innovative imaging techniques for advancing cancer diagnosis and treatment", Cancers, 16(14), 2607, 2024.
• T. Yousaf, G. Dervenoulas and M. Politis, "Advances in MRI methodology", International Review of Neurobiology, 141, 31-76, 2018.
• Z. Zhou and Z. R. Lu, "Gadolinium-based contrast agents for magnetic resonance cancer imaging", Wiley Interdisciplinary Reviews: Nanomedicine and Nanobiotechnology, 5(1), 1-18, 2013.
• Y. Yu, D. H. Lee, S. L. Peng, K. Zhang, Y. Zhang, S. Jiang, et al., "Assessment of glioma response to radiotherapy using multiple MRI biomarkers with manual and semiautomated segmentation algorithms", Journal of Neuroimaging, 26(6), 626-634, 2016.
• D. Gala, H. Behl, M. Shah and A. N. Makaryus, "The role of artificial intelligence in improving patient outcomes and future of healthcare delivery in cardiology: A narrative review of the literature", Healthcare (Basel), 12(4), 481, 2024.
• S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 4th ed., Pearson, 2021.
• G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken and C. I. Sánchez, "A survey on deep learning in medical image analysis", Medical Image Analysis, 42, 60-88, 2017.
• I. Hammad and K. El-Sankary, "Impact of approximate multipliers on VGG deep learning network", IEEE Access, 6, 60438-60444, 2018.
• S. H. Wang and Y. D. Zhang, "DenseNet-201-based deep neural network with composite learning factor and precomputation for multiple sclerosis classification", ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 16(2s), 1-19, 2020.
• R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018.
• L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, O. Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidie and L. Farhan, “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions”, Journal of Big Data, 8(1), 53, 2021.
• Y. N. Kuan, K. M. Goh and L. L. Lim, "Systematic review on machine learning and computer vision in precision agriculture: Applications, trends, and emerging techniques", Engineering Applications of Artificial Intelligence, 148, 110401, 2025.
• D. Dablain, K. N. Jacobson, C. Bellinger, M. Roberts and N. M. Chawla, "Understanding CNN fragility when learning with imbalanced data", Machine Learning, 113, 4785-4810, 2024.
• S. M. Polisetty, "AI-driven diagnosis and treatment recommendation in healthcare: A hybrid deep learning framework", International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2025.
• A. B. Sicilia, X. Zhao, A. Sosnovskikh and S. J. Hwang, "PAC Bayesian performance guarantees for deep (stochastic) networks in medical imaging", Medical Image Computing and Computer-Assisted Intervention – MICCAI 2021, 12903, 560-570, 2021.
• X. Jiang, Z. Hu, S. Wang and Y. Zhang, "Deep learning for medical image-based cancer diagnosis", Cancers, 15(14), 3608, 2023.
• A. K. Mandle, S. Sahu and G. P. Gupta, "CNN-based deep learning technique for the brain tumor identification and classification in MRI images", International Journal of Software Science and Computational Intelligence, 14(1), 1-16, 2022.
• S. S. A. Khan, A. Prova and U. Acharjee, "MRI-based brain tumor image classification using CNN", Asian Journal of Research in Computer Science, 15(13), 1-10, 2023
• M. A. Amou, K. Xia, S. Kamhi, and M. Mouhafid, "A Novel MRI Diagnosis Method for Brain Tumor Classification Based on CNN and Bayesian Optimization," Healthcare, 10 (3), 494, 2022.
• J. Cheng, W. Yang, M. Huang, W. Huang, J. Jiang, Y. Zhou, R. Yang, J. Zhao, Y. Feng, Q. Feng and W. Chen, “Retrieval of brain tumors by adaptive spatial pooling and Fisher vector representation”, PloS One, 11(6), e0157112, 2016.
• M. I. Sharif, J. P. Li, M. A. Khan and M. A. Saleem, "Active deep neural network features selection for segmentation and recognition of brain tumors using MRI images", Pattern Recognition Letters, 129, 181-189, 2020.
• R. A. Zeineldin, M. E. Karar, O. Burgert and F. Mathis-Ullrich, "Multimodal CNN networks for brain tumor segmentation in MRI: A BraTS 2022 challenge solution", International MICCAI Brainlesion Workshop, 13409, 127-137, 2022.
• M. Attique, G. Gilanie, M. S. Mehmood, M. S. Naweed, M. Ikram, J. A. Kamran and A. Vitkin, “Colorization and automated segmentation of human T2 MR brain images for characterization of soft tissues”, PloS One, 7(3), e33616, 2012.
• A. Biswas, S. Abedin, and M. A. Kabir, "Moving Object Detection Using Ultrasonic Radar with Proper Distance, Direction, and Object Shape Analysis," Journal of Information Systems Engineering and Business Intelligence, 6, 2, 2020.
• K. Neamah, F. Mohamed, S. R. Waheed, W. H. M. Kurdi, A. Y. Taha and K. A. Kadhim, "Utilizing deep improved ResNet50 for brain tumor classification based MRI", IEEE Open Journal of the Computer Society, 5, 446-456, 2024.
• H. A. Munira and M. S. Islam, "Hybrid deep learning models for multi-classification of tumour from brain MRI", Journal of Information Systems Engineering and Business Intelligence, 8(2), 162-174, 2022.
• P. Pilaoon, A. Narkthewan, N. Wadlom, R. Varakulsiripunth, K. Hamamoto and N. Maneerat, "Glioma brain tumor classification using transfer learning", Proceedings of the 2024 10th International Conference on Engineering, Applied Sciences, and Technology (ICEAST), 17-22, 2024.
• P. Pilaoon, A. Narkthewan, N. Wadlom, R. Varakulsiripunth, K. Hamamoto and N. Maneerat, "Glioma brain tumor classification using transfer learning", Proceedings of the 10th International Conference on Engineering, Applied Sciences and Technology (ICEAST), 17–22, 2024.
• M. A. Khan, M. Z. Hussain, S. Mehmood, M. F. Khan, M. Ahmad, T. Mazhar, T. Shahzad, and M. M. Saeed, “Transfer learning for accurate brain tumor classification in MRI: A step forward in medical diagnostics,” Discover Oncology, vol. 16, no. 1040, 2025.
• A. Sevinc, M. Uçan and B. Kaya, "A distillation approach to transformer-based medical image classification with limited data", Diagnostics, 15(7), 929, 2025.