Multiple Attention-Based Deep Learning Model for MRI Captioning

Abstract

In recent years, the use of artificial intelligence in medicine, as in many other fields, has begun to increase considerably. Creating magnetic resonance (MR) reports manually by medical doctors is a very difficult, time-consuming, and potentially error-prone process. In order to address these problems, a deep learning-based image captioning model is proposed in this study to automatically generate reports from brain MRIs. In the developed model, image processing, natural language processing, and deep learning methods are used together to produce text for the content and diagnoses in the medical image. First of all, pre-processing, such as rotating at random angles, changing size, cropping, changing brightness and contrast, adding shadows, and mirroring, were performed for MR images. Then, a model that generates reports was developed by utilizing the Bootstrapping Language Image Pre-Training (BLIP) model and the transformer architecture of the model. The experimental studies showed that the proposed model had successful results; the produced reports were highly similar to the original reports and could be used as a supplementary tool in medicine.

Keywords

• Brain MRI
• BLIP
• Deep Learning
• Medical Image Captioning

MR Altyazılama için Çoklu Dikkat Tabanlı Derin Öğrenme Modeli

Abstract

Son yıllarda birçok alanda olduğu gibi sağlık alanında da yapay zeka kullanımı oldukça artmaya başlamıştır. Manyetik rezonans (MR) raporlarının manuel olarak tıp hekimleri tarafından oluşturulması oldukça zor, uzun zaman alan ve hatalı olma olasılığı yüksek bir süreçtir. Bu problemleri adreslemek amacıyla, bu çalışmada beyin MR görüntülerinden otomatik rapor üretecek derin öğrenme tabanlı görüntü altyazılama modeli önerilmiştir. Geliştirilen modelde, görüntü işleme, doğal dil işleme ve derin öğrenme yöntemleri birlikte kullanılarak tıbbi görüntüdeki içerik ve tanılara yönelik metin üretilmektedir. Öncelikle MR görüntüleri için, rastgele açılarla döndürme, boyut değiştirme, kırpma, parlaklık ve kontrast değiştirme, gölge ekleme ve aynalama gibi ön işlemler yapılmıştır. Ardından Bootstrapping Language Image Pre-Training (BLIP) modeli ve modelin transformer mimarisinden faydalanılarak rapor üreten bir model geliştirilmiştir. Yapılan deneysel çalışmalarda, geliştirilen modelin farklı metrikler için başarılı sonuçlar verdiği, üretilen raporların orijinal raporlara yüksek oranda benzer olduğu ve tıp alanında yardımcı öneri sistemi olarak kullanılabileceği görülmüştür.

Keywords

• MR
• BLIP
• Derin Öğrenme
• Görüntü Altyazılama

References

1. [1] Ravinder, P., & Srinivasan, S. (2024). Automated Medical Image Captioning with Soft Attention-Based LSTM Model Utilizing YOLOv4 Algorithm.
2. [2] Aggarwal, A. K. Revealing AI-Driven Chest X-Ray Image Captioning Using Blip Transformer.
3. [3] Wang, Y., Lin, Z., Xu, Z., Dong, H., Luo, J., Tian, J., ... & He, Z. (2024). Trust it or not: Confidence-guided automatic radiology report generation. Neurocomputing, 127374.
4. [4] Boag, W., Hsu, T. M. H., McDermott, M., Berner, G., Alesentzer, E., & Szolovits, P. (2020, April). Baselines for chest x-ray report generation. In Machine learning for health workshop (pp. 126-140). PMLR.
5. [5] Liu, F., Yin, C., Wu, X., Ge, S., Zou, Y., Zhang, P., Zou, Y., & Sun, X. (2023). Contrastive attention for automatic chest X-ray report generation. arXiv. https://arxiv.org/abs/2106.06965.
6. [6] Lovelace, J., & Mortazavi, B. (2020, November). Learning to generate clinically coherent chest X-ray reports. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 1235-1243).
7. [7] Yang, S., Wu, X., Ge, S., Zhou, S. K., & Xiao, L. (2022). Knowledge matters: Chest radiology report generation with general and specific knowledge. Medical image analysis, 80, 102510
8. [8] Yang, S., Wu, X., Ge, S., Zheng, Z., Zhou, S. K., & Xiao, L. (2023). Radiology report generation with a learned knowledge base and multi-modal alignment. Medical Image Analysis, 86, 102798.

9. [9] Pelka, O., Koitka, S., Rückert, J., Nensa, F., & Friedrich, C. M. (2018). Radiology objects in context (roco): a multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3 (pp. 180-189). Springer International Publishing.
10. [10] Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." Proceedings of the 40th International Conference on Machine Learning, PMLR 202:19730-19742, 2023.
11. [11] Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi. "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation." Proceedings of the 39th International Conference on Machine Learning, PMLR 162:12888-12900, 2022.
12. [12] Lemons, S., Linares López, C., Holte, R., & Ruml, W. (2022). Beam search: Faster and monotonic. Proceedings of the International Conference on Automated Planning and Scheduling, 32(1), 222-230. https://doi.org/10.1609/icaps.v32i1.19805.
13. [13] Zeng, X., Wen, L., Xu, Y., & Ji, C. (2020). Generating diagnostic report for medical image by high-middle-level visual information incorporation on double deep learning models. Computer methods and programs in biomedicine, 197, 105700.
14. [14] Barbella, M., & Tortora, G. (2022). Rouge metric evaluation for text summarization techniques. Available at SSRN 4120317.
15. [15] Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
16. [16] https://www.imageclef.org/2024/medical/caption
17. [17] Liu, G., Hsu, T. M. H., McDermott, M., Boag, W., Weng, W. H., Szolovits, P., & Ghassemi, M. (2019, October). Clinically accurate chest x-ray report generation. In Machine Learning for Healthcare Conference (pp. 249-269). PMLR.