Classification of Breast Lesions via Active Thermograms with the Help of Deep Learning

Abstract

In recent years, artificial intelligence studies, which have developed in parallel with computer hardware, have helped increase patient survival by preventing possible metastasis with early diagnosis of clinicians. There are many studies in literature that carry out diagnosis of cancer in the clinic. In these studies, machine learning and deep learning applications are frequently applied for cancer classification. Similarly, in this study, diagnosis of breast cancer over thermal images with the help of deep learning methods was discussed. The images used in the study were taken open access made available in the DMR-IR dataset. Some preprocessing was done on the images before manual and automatic segmentation methods were applied to segment the breast regions on images. In the manual segmentation process, the mask of the breast regions whose localization information was recorded with VIA was created and the segmentation was performed by subtracting the mask from the original image. In the automatic segmentation process, segmentation was done using Mask R-CNN and U-NET techniques. Segmentation performance analysis was performed for these two methods and classification operations were done with Mask R-CNN, which realized 0.9896 accuracy, 0.9413 Dice and 0.8900 Jaccard. Breast cancer classification was carried out using seven pre-trained architectures (InceptionV3, MobileNet, MobileNetV2, ResNet50, VGG16, VGG19 and Xception) with thermograms consisting of images segmented by manual and Mask-RCNN methods. As a result, MobileNet and InceptionV3 architectures provided the 100% classification success in test data accuracy, precision, sensitivity and F1 Score.

Keywords

• Thermal imaging
• Breast lesion
• Classification
• Segmentation
• Mask R-CNN
• U-Net
• Transfer learning

Derin Öğrenme Yardımıyla Aktif Termogramlar Üzerinden Meme Lezyonlarının Sınıflandırması

Abstract

Son yıllarda bilgisayar donanımları ile paralel olarak gelişim gösteren yapay zeka çalışmaları klinikte uzmanların erken teşhis ile olası metastazın önüne geçerek hasta sağ kalımını artırmaktadır. Literatürde klinikte kanser teşhisini gerçekleştiren çokça çalışma mevcuttur. Bu çalışmalarda, kanser sınıflandırmasının yapılması için makine öğrenmesi ve derin öğrenme uygulamaları sıklıkla uygulanmaktadır. Benzer şekilde çalışmada termal meme görüntüleri üzerinden derin öğrenme yöntemleri ile meme kanseri teşhisi ele alınmıştır. Çalışmada kullanılan görüntüler açık erişim olarak sunulan DMR-IR veri setinden alınmıştır. Veri setinden alınan görüntüler üzerinde bazı önişlemler yapılmış, ardından meme bölgelerinin bölütlenmesi için manuel ve otomatik olmak üzere iki farklı bölütleme metodu uygulanmıştır. Manuel bölütleme işleminde, VIA ile lokalizasyon bilgisi kaydedilen meme bölgelerinin maskesi oluşturup orijinal görüntüden çıkarılarak bölütleme gerçekleştirilmiştir. Otomatik bölütleme işleminde ise Mask R-CNN ve U-NET ile bölütleme yapılmıştır. Bu iki metot için bölütleme performans analizi yapılmış ve 0.9896 doğruluk, 0.9413 Dice ve 0.8900 Jaccard değerini gerçekleştiren Mask R-CNN ile sınıflandırma işlemleri çalışılmıştır. Manuel ve Mask-RCNN metodu ile bölütlenen görüntülerden oluşan termogramlar ile ön eğitimli yedi farklı (InceptionV3, MobileNet, MobileNetV2, ResNet50, VGG16, VGG19 ve Xception) mimari kullanılarak meme kanseri sınıflandırması gerçekleştirilmiştir. Sonuç olarak test verilerinde %100 sınıflandırma başarısını doğruluk, kesinlik, duyarlılık ve F1 Skoru ile MobileNet ve InceptionV3 mimarileri sağlamıştır.

Keywords

• Termal görüntüleme
• Meme lezyonu
• Sınıflandırma
• Bölütleme
• Mask R-CNN
• U-Net
• Transfer öğrenme

References

1. N. Harbeck and M. Gnant, "Early breast cancer: treatment concepts and biology," J. Breast Cancer, vol. 18, no. 4, pp. 303-312, 2016.
2. H. Sung et al., "Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries," CA: a cancer journal for clinicians, vol. 71, no. 3, pp. 209-249, 2021.
3. M. d. F. O. Baffa and L. G. Lattari, "Convolutional neural networks for static and dynamic breast infrared imaging classification," in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), 2018: IEEE, pp. 174-181.
4. C. N. Karim, O. Mohamed, and T. Ryad, "A new approach for breast abnormality detection based on thermography," Medical Technologies Journal, vol. 2, no. 3, pp. 245-254, 2018.
5. J. Zuluaga-Gomez, Z. Al Masry, K. Benaggoune, S. Meraghni, and N. Zerhouni, "A CNN-based methodology for breast cancer diagnosis using thermal images," Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, vol. 9, no. 2, pp. 131-145, 2021.
6. B. Alafi, "Artificial Intelligence and Deep Learning," THE JOURNAL OF COGNITIVE SYSTEMS, vol. 4, no. 2, pp. 57-61, 2019. [Online]. Available: https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/neural_networks.html.
7. M. Goyal, T. Knackstedt, S. Yan, and S. Hassanpour, "Artificial intelligence-based image classification methods for diagnosis of skin cancer: Challenges and opportunities," Comput Biol Med, vol. 127, p. 104065, Dec 2020, doi: 10.1016/j.compbiomed.2020.104065.
8. H. Benbrahim, H. Hachimi, and A. Amine, "Deep convolutional neural network with tensorflow and keras to classify skin cancer images," Scalable Computing, vol. 21, no. 3, pp. 379-389, 2020, doi: 10.12694:/scpe.v21i3.1725.

9. D. A. Shoieb, S. M. Youssef, and W. M. Aly, "Computer-Aided Model for Skin Diagnosis Using Deep Learning," Journal of Image and Graphics, pp. 122-129, 2016, doi: 10.18178/joig.4.2.122-129.
10. R. Roslidar et al., "A review on recent progress in thermal imaging and deep learning approaches for breast cancer detection," IEEE Access, vol. 8, pp. 116176-116194, 2020.
11. H. Dhahri, E. Al Maghayreh, A. Mahmood, W. Elkilani, and M. Faisal Nagi, "Automated breast cancer diagnosis based on machine learning algorithms," Journal of healthcare engineering, vol. 2019, 2019.
12. N. I. Yassin, S. Omran, E. M. El Houby, and H. Allam, "Machine learning techniques for breast cancer computer aided diagnosis using different image modalities: A systematic review," Computer methods and programs in biomedicine, vol. 156, pp. 25-45, 2018.
13. Y. Jiménez-Gaona, M. J. Rodríguez-Álvarez, and V. Lakshminarayanan, "Deep-learning-based computer-aided systems for breast cancer imaging: a critical review," Applied Sciences, vol. 10, no. 22, p. 8298, 2020.
14. L. Abdelrahman, M. Al Ghamdi, F. Collado-Mesa, and M. Abdel-Mottaleb, "Convolutional neural networks for breast cancer detection in mammography: A survey," Computers in Biology and Medicine, p. 104248, 2021, doi: 10.1016/j.compbiomed.2021.104248.
15. G. Schaefer, M. Závišek, and T. Nakashima, "Thermography based breast cancer analysis using statistical features and fuzzy classification," Pattern recognition, vol. 42, no. 6, pp. 1133-1137, 2009.
16. U. R. Acharya, E. Y.-K. Ng, J.-H. Tan, and S. V. Sree, "Thermography based breast cancer detection using texture features and support vector machine," Journal of medical systems, vol. 36, no. 3, pp. 1503-1510, 2012.
17. M. R. K. Mookiah, U. R. Acharya, and E. Ng, "Data mining technique for breast cancer detection in thermograms using hybrid feature extraction strategy," Quantitative InfraRed Thermography Journal, vol. 9, no. 2, pp. 151-165, 2012.
18. N. Golestani, M. EtehadTavakol, and E. Ng, "Level set method for segmentation of infrared breast thermograms," EXCLI journal, vol. 13, p. 241, 2014.
19. M. Milosevic, D. Jankovic, and A. Peulic, "Thermography based breast cancer detection using texture features and minimum variance quantization," EXCLI journal, vol. 13, p. 1204, 2014.
20. S. Pramanik, D. Bhattacharjee, and M. Nasipuri, "Wavelet based thermogram analysis for breast cancer detection," in 2015 international symposium on advanced computing and communication (ISACC), 2015: IEEE, pp. 205-212.
21. F. J. Fernández-Ovies, E. S. Alférez-Baquero, E. J. de Andrés-Galiana, A. Cernea, Z. Fernández-Muñiz, and J. L. Fernández-Martínez, "Detection of breast cancer using infrared thermography and deep neural networks," in International Work-Conference on Bioinformatics and Biomedical Engineering, 2019: Springer, pp. 514-523.
22. S. Tello-Mijares, F. Woo, and F. Flores, "Breast cancer identification via thermography image segmentation with a gradient vector flow and a convolutional neural network," Journal of healthcare engineering, vol. 2019, 2019.
23. Y. Liang, R. He, Y. Li, and Z. Wang, "Simultaneous segmentation and classification of breast lesions from ultrasound images using mask R-CNN," in 2019 IEEE International Ultrasonics Symposium (IUS), 2019: IEEE, pp. 1470-1472.
24. J.-Y. Chiao, K.-Y. Chen, K. Y.-K. Liao, P.-H. Hsieh, G. Zhang, and T.-C. Huang, "Detection and classification the breast tumors using mask R-CNN on sonograms," Medicine, vol. 98, no. 19, 2019.
25. M. A. Farooq and P. Corcoran, "Infrared Imaging for Human Thermography and Breast Tumor Classification using Thermal Images," Letterkenny, Ireland, 11-12 June 2020 2020: IEEE, doi: 10.1109/ISSC49989.2020.9180164.
26. H. Ghayoumi Zadeh, A. Fayazi, B. Binazir, and M. Yargholi, "Breast Cancer Diagnosis Based on Feature Extraction Using Dynamic Models of Thermal Imaging and Deep Autoencoder Neural Networks," Journal of Testing and Evaluation, vol. 49, no. 3, 2021, doi: 10.1520/jte20200044.
27. S. Civilibal, K. K. Cevik, and A. Bozkurt, "A deep learning approach for automatic detection, segmentation and classification of breast lesions from thermal images," Expert Systems with Applications, vol. 212, p. 118774, 2023.
28. L. Silva et al., "A new database for breast research with infrared image," Journal of Medical Imaging and Health Informatics, vol. 4, no. 1, pp. 92-100, 2014.
29. A. Dutta and A. Zisserman, "The VIA annotation software for images, audio and video," in Proceedings of the 27th ACM international conference on multimedia, 2019, pp. 2276-2279.
30. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
31. K. O'Shea and R. Nash, "An introduction to convolutional neural networks," arXiv preprint arXiv:1511.08458, 2015.
32. S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: towards real-time object detection with region proposal networks," IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 6, pp. 1137-1149, 2016.
33. K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
34. Y. Weng, T. Zhou, Y. Li, and X. Qiu, "Nas-unet: Neural architecture search for medical image segmentation," IEEE Access, vol. 7, pp. 44247-44257, 2019.
35. O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in International Conference on Medical image computing and computer-assisted intervention, 2015: Springer, pp. 234-241.
36. U. Snekhalatha and K. Sangamithirai, "Computer aided diagnosis of obesity based on thermal imaging using various convolutional neural networks," Biomedical Signal Processing and Control, vol. 63, p. 102233, 2021.
37. M. B. Lopez, C. R. del-Blanco, and N. Garcia, "Detecting exercise-induced fatigue using thermal imaging and deep learning," in 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), 2017: IEEE, pp. 1-6.
38. M. Aslanyürek and A. Mesut, "Kümeleme Performansını Ölçmek için Yeni Bir Yöntem ve Metin Kümeleme için Değerlendirmesi," Avrupa Bilim ve Teknoloji Dergisi, no. 27, pp. 53-65, 2021.
39. O. Bilginer, B. Tunga, and R. M. Demirer, "Classification of skin lesions using convolutional neural networks," Pamukkale Univ Muh Bilim Derg, vol. 28, no. 2, pp. 208-214, 2022, doi: 10.5505/pajes.2021.68700.