Radyoloji Raporlarına Sonuç Kısmı Üretmede Geniş Dil Modellerinin Başarısı

Abstract

Amaç: Doğal dil işleme metrikleri kullanılarak Türkçe kontrastsız toraks bilgisayarlı tomografi (BT) bulgularından otomatik olarak sonuç bölümü oluşturulmasında ChatGPT 5.5 ve Gemini 3.5 Flash'ın performansını değerlendirmek ve karşılaştırmak. Gereç ve Yöntemler: Bu retrospektif çalışmada, radyologlar tarafından yazılmış 50 kontrastsız toraks BT raporunun "Bulgular" bölümleri, API aracılığıyla ChatGPT ve Gemini tarafından işlenmiştir. Standartlaştırılmış, kural tabanlı bir prompt kullanılmıştır. Büyük dil modelleri (LLM) tarafından oluşturulan sonuç bölümleri; anlamsal benzerlik için Türkçeye uyarlanmış BERTScore, yapısal örtüşme için ROUGE-L ve sadeliği değerlendirmek için metin uzunluk oranı kullanılarak orijinal uzman radyolog sonuçları ile karşılaştırılmıştır. Bulgular: ChatGPT, Gemini'ye kıyasla istatistiksel olarak anlamlı derecede daha yüksek genel BERTScore F1 metriği (0.6780 ± 0.0712 vs 0.6427 ± 0.0881, p < 0.001) ve kesinlik (precision) (0.6178 ± 0.0757 vs 0.5612 ± 0.0864, p < 0.001) göstermiştir. Recall (duyarlılık) değerleri iki model arasında benzer bulunmuştur (p = 0.9719). ROUGE-L F1 ile değerlendirilen yapısal benzerlikte anlamlı bir fark izlenmemiştir (p = 0.5656). Her iki model de orijinal insan raporlarından anlamlı derecede daha uzun sonuç bölümleri üretmiştir (oran > 1.0); ancak ChatGPT'nin metin uzunluk oranı anlamlı derecede daha düşük bulunmuş olup, bu durum Gemini'ye kıyasla daha öz ve kısa bir çıktı ürettiğini göstermektedir (2.4287 vs 3.0564, p < 0.001). Sonuç: Her iki model de klinik özetleri sentezleme kapasitesi göstermekte olup, ChatGPT Türkçe radyoloji metinleri için üstün anlamsal kesinlik (precision) sunmaktadır. Bununla birlikte, insan radyologların kompakt raporlama stiline kıyasla her iki modelin de detaylı ve uzun üslubu ile anlamsal ve yapısal skorlardaki nispeten düşük performansları, klinik iş akışlarına güvenli ve verimli bir entegrasyon için insan denetimli bir yaklaşımı zorunlu kılmaktadır.

Keywords

• dil modelleri
• yapay zeka
• radyoloji
• rapor
• doğal dil işleme

Supporting Institution

N/A

Ethical Statement

Bu çalışma için etik kurul onayı, 11.06.2026 tarihinde İzmir Kâtip Çelebi Sağlık Araştırmaları Etik Kurulu'ndan alınmıştır (Karar No: 0513).

Thanks

N/A

Evaluation of Large Language Models for Radiology Report Conclusion Generation

Abstract

Purpose: To evaluate and compare the performance of ChatGPT 5.5 and Gemini 3.5 Flash in automatically generating conclusion sections from Turkish non-contrast thorax computed tomography (CT) findings using natural language processing metrics. Materials and Methods: In this retrospective study, "Findings" sections from 50 consecutive radiologist-authored non-contrast thorax CT reports were processed by ChatGPT and Gemini via API. A standardized, rule-based prompt was utilized. The LLM-generated conclusions were quantitatively compared to the original expert radiologist conclusions (reference standard) using a Turkish-adapted BERTScore for semantic similarity, ROUGE-L for structural overlap, and a text length ratio to assess clinical conciseness. Results: ChatGPT demonstrated a significantly higher overall BERTScore F1 metric (0.6780 ± 0.0712 vs. 0.6427 ± 0.0881, p < 0.001) and precision (0.6178 ± 0.0757 vs. 0.5612 ± 0.0864, p < 0.001) compared to Gemini. Recall was comparable between the two models (p = 0.9719). Structural similarity via ROUGE-L F1 showed no significant difference (p = 0.5656). Both models generated conclusion sections significantly longer than the original human reports (ratio > 1.0); however, the text length ratio of ChatGPT was significantly lower, indicating a more concise output compared to Gemini (2.4287 vs. 3.0564, p < 0.001). Conclusion: Both models demonstrate the capacity to synthesize clinical summaries, with ChatGPT offering superior semantic precision for Turkish radiology texts. However, the verbosity of both models compared to the compact reporting style of human radiologists, and comparably lower performance on semantic and structural scores necessitates a human-in-the-loop approach for safe and efficient integration into clinical workflows.

Keywords

• Language models
• artificial intelligence
• radiology
• report
• natural language processing

Supporting Institution

N/A

Ethical Statement

Ethical approval for this study was obtained from the İzmir Kâtip Çelebi Health Research Institutional Review Board on June 11, 2026 (IRB #: 0513). The institutional review board waived the requirement for informed consent due to the retrospective nature of the study and the use of non-identifiable data.

Thanks

N/A

References

1. 1. Flanders AE, Lakhani P. Radiology reporting and communications: a look forward. Neuroimaging Clin N Am. 2012;22:477-96. doi: 10.1016/j.nic.2012.04.009.
2. 2. Bosmans JML, Weyler JJ, De Schepper AM, Parizel PM. The radiology report as seen by radiologists and referring clinicians: results of the COVER and ROVER surveys. Radiology. 2011;259:184-95. doi: 10.1148/radiol.10101045.
3. 3. Reiner BI, Knight N, Siegel EL. Radiology reporting, past, present, and future: the radiologist's perspective. J Am Coll Radiol. 2007;4:313-9. doi: 10.1016/j.jacr.2007.01.015.
4. 4. Hartung MP, Bickle IC, Gaillard F, Kanne JP. How to create a great radiology report. Radiographics. 2020;40:1658-70. doi: 10.1148/rg.2020200020.
5. 5. Lukaszewicz A, Uricchio J, Gerasymchuk G. The art of the radiology report: practical and stylistic guidelines for perfecting the conveyance of imaging findings. Can Assoc Radiol J. 2016;67:318-21. doi: 10.1016/j.carj.2016.03.001.
6. 6. Salbas A, Kul Baysan E. Assessment of large language models in musculoskeletal radiological anatomy: a comparative study with radiologists. Jt Dis Relat Surg. 2026;37:190-9. doi: 10.52312/jdrs.2026.2436.
7. 7. Güzel HE, Oleaga L, Koç AM, Junquero V, Merino C. Large language models solving the European Diploma in Radiology: a comparative evaluation. Acad Radiol. 2026;33:1871-8. doi: 10.1016/j.acra.2026.01.040
8. 8. Büyüktoka RE, Surucu M, Erekli Derinkaya PB, et al. Applying large language model for automated quality scoring of radiology requisitions using a standardized criteria. Eur Radiol. Epub ahead of print 2025. doi: 10.1007/s00330-025-11933-2.

9. 9. Karabulut A, Kantarcı S, Dağlı U, et al. Patients ask, AI answers: an evaluation of ChatGPT-4o responses to the most frequently googled questions on laparoscopic hysterectomy. Anatol J Gen Med Res. 2026;36:84-9. doi: 10.4274/anatoljmed.2026.97759.
10. 10. Bozer A, Pekçevik Y. Comparative evaluation of large language models in explaining radiology reports: expert assessment of readability, understandability, and communication features. Insights Imaging. 2025;16:232. doi: 10.1186/s13244-025-02121-3.
11. 11. Alabed S, Anderson A, Maiter A, et al. Large language models for simplifying radiology reports: a systematic review and meta-analysis of patient, public, and clinician evaluations. Lancet Digit Health. 2026;8:100960. doi: 10.1016/j.landig.2025.100960.
12. 12. Zhang L, Liu M, Wang L, et al. Constructing a large language model to generate impressions from findings in radiology reports. Radiology. 2024;312:e240885. doi: 10.1148/radiol.240885.
13. 13. Salbas A, Tekel GM, Büyüktoka AD, Büyüktoka RE, Koç AM, Çilengir AH. Evaluation of artificial intelligence-generated information about abdominal ultrasonography. Istanbul Med J. 2026;27(2). doi: 10.4274/imj.galenos.2026.95826.
14. 14. Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: evaluating text generation with BERT. Available from: http://arxiv.org/abs/1904.09675. 2020.
15. 15. Ganesan K. ROUGE 2.0: updated and improved measures for evaluation of summarization tasks. Available from: http://arxiv.org/abs/1803.01937. 2018.
16. 16. dbmdz, Bayerische Staatsbibliothek. BERTurk: bert-base-turkish-cased [Internet]. Available from: https://huggingface.co/dbmdz/bert-base-turkish-cased. Accessed July 1, 2026.
17. 17. Hasani AM, Singh S, Zahergivar A, et al. Evaluating the performance of Generative Pre-trained Transformer-4 (GPT-4) in standardizing radiology reports. Eur Radiol. 2024;34:3566-74. doi: 10.1007/s00330-023-10384-x.
18. 18. Li J, Lin Y, Zhao P, et al. Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT). BMC Med Inform Decis Mak. 2022;22:200. doi: 10.1186/s12911-022-01946-y.
19. 19. Olthof AW, Shouche P, Fennema EM, et al. Machine learning based natural language processing of radiology reports in orthopaedic trauma. Comput Methods Programs Biomed. 2021;208:106304. doi: 10.1016/j.cmpb.2021.106304.
20. 20. Lyu M, Zhang M, Peng C, Lu JL, Chen Z, Wu Y. Improving automatic summarization of radiology reports through mid-training of large language models. Available from: http://arxiv.org/abs/2603.19275. 2026.
21. 21. Pourgholamali S, Patti E, Aliberti A. A modular hybrid transformer framework for trustworthy radiology report summarization. In: 2026 IEEE Conference on Artificial Intelligence (CAI). IEEE; 2026:1102-7.
22. 22. Sultan T, Rony MAT, Islam MS, Alshathri S, El-Shafai W. SumGPT: a multimodal framework for radiology report summarization to improve clinical performance. IEEE Access. 2025;13:15929-45. doi: 10.1109/ACCESS.2025.3528335.