Multimodal Büyük Dil Modellerinin BT’de Bone-RADS Klasifikasyonundaki Tanısal Performansı

Öz

Çalışmamızın amacı multimodal büyük dil modellerinin (MBDM) BT görüntülerinde tespit edilen soliter kemik lezyonlarına Bone-RADS kategorileri atamadaki performanslarını değerlendirmektir. Hastanemizin PACS’ı soliter kemik lezyonu içeren BT tetkikleri için taranmıştır (Ağustos 2024-Ağustos 2025). Her lezyon için bir kas-iskelet radyoloğu tarafından kitleyi en iyi temsil eden bir kesit seçilmiş ve lezyonlara birer referans Bone-RADS skoru atanmıştır. Daha sonra bir abdominal radyolog, ChatGPT 5 ve Gemini 2.5 Pro aynı vakaları kategorilemiştir. Doğruluk, doğru şekilde kategorize edilen Bone-RADS 1 ve 4 vakaları olarak tanımlanmış ve McNemar testi kullanılarak karşılaştırılmıştır. Referansla uyum, ağırlıklı Cohen κ katsayısı kullanılarak değerlendirilmiş ve bootstrap yöntemi ile karşılaştırılmıştır. Referans kategorileri şu şekilde belirlenmiştir: Bone-RADS 1, n=23; 2, n=4; 3, n=0; 4, n=23. Doğruluk, radyolog için %84,8 (39/46), Gemini için %78,3 (36/46) ve ChatGPT için %65,2 (30/46) olarak bulunmuştur. Radyoloğun, ChatGPT'den daha iyi performans gösterdiği (p=0,012); radyolog ile Gemini (p=0,604) ve Gemini ile ChatGPT (p=0,360) arasındaki farkların anlamlı olmadığı görülmüştür. Radyolog, referans standardı ile en yüksek uyumu elde etmiş (κ = 0,715, %95 GA: [0,543-0,887]), bunu Gemini (κ = 0,542, %95 GA: [0,313-0,770]) ve ChatGPT (κ = 0,292, %95 GA: [0,104-0,479]) izlemiştir. Bootstrap ile yapılan karşılaştırmalar, radyoloğun κ değerinin ChatGPT'den daha yüksek olduğunu göstermiş (%95 GA: 0,140-0,675), ancak radyolog ile Gemini (%95 GA: −0,113-0,434) ve Gemini ile ChatGPT (%95 GA: −0,041-0,522) arasındaki fark anlamlı bulunmamıştır. Sonuç olarak genel amaçlı MBDM'ler henüz Bone-RADS kategorizasyonu için eğitimli radyologların yerini tutabilecek durumda görünmemekle beraber bu modellerin günlük pratikte radyologlara yardımcı olabileceği düşünülmektedir.

Anahtar Kelimeler

• MBDM
• BDM
• Bone-RADS
• ChatGPT
• Gemini

Diagnostic Performance of Multimodal Large Language Models in Assigning Bone-RADS Categories on CT

Öz

The aim of the study is to assess the performance of multimodal large language models (MLLMs) in assigning Bone-RADS categories to bone lesions identified on CT images. An MSK radiologist selected one representative slice for 50 bone lesions seen on CT studies and assigned reference Bone-RADS categories using clinical records. Three raters categorized each case: an abdominal radiologist, OpenAI ChatGPT 5, and Google Gemini 2.5 Pro. Accuracy was defined as the correctly labeled Bone-RADS 1 and 4 cases and compared using McNemar test. Agreement with the reference was assessed using weighted Cohen’s κ with 95% CIs; pairwise κ differences were tested via bootstrap. Reference categories were Bone-RADS 1, n=23; 2, n=4; 3, n=0; 4, n=23. Accuracy was 84.8% (39/46) for the radiologist, 78.3% (36/46) for Gemini, and 65.2% (30/46) for ChatGPT. The radiologist outperformed ChatGPT (p=0.012); differences between the radiologist vs Gemini (p=0.604) and Gemini vs ChatGPT (p=0.360) were not significant. The radiologist achieved the highest agreement with the reference standard (κ = 0.715, 95% CI: [0.543-0.887]), followed by Gemini (κ = 0.542, 95% CI: [0.313-0.770]) and ChatGPT (κ = 0.292, 95% CI: [0.104-0.479]). Bootstrap comparisons showed that the radiologist’s κ was higher than ChatGPT’s (95% CI for difference, 0.140-0.675), while radiologist vs Gemini (−0.113-0.434) and Gemini vs ChatGPT (−0.041-0.522) were not significant. In conclusion, general-purpose MLLMs cannot yet replace trained radiologists for Bone-RADS classification, though they may still aid routine clinical practice.

Anahtar Kelimeler

• MLLMs
• LLMs
• Bone-RADS
• ChatGPT
• Gemini

Kaynakça

1. 1.Blackburn CW, Richardson SM, Devita RR, et al. What Is the Prevalence of Clinically Important Findings AmongIncidentally Found Osseous Lesions? Clin Orthop Relat Res. 2023;481(10):1993-2002. doi:10.1097/CORR.0000000000002630
2. 2. Salunke AA, Nandy K, Puj K, et al. A proposed “Radiological Evaluation Score for Bone Tumors” (REST): An objective system for assessment of a radiograph in patients with suspected bone tumor. Musculoskelet Surg. 2022;106(4):371-382. doi:10.1007/S12306-021-00711-0
3. 3. Caracciolo JT, Ali S, Chang CY, et al. Bone Tumor RiskStratification and Management System: A Consensus Guideline from the ACR Bone Reporting and Data System Committee. Journal of the American College of Radiology. 2023;20(10):1044-1058. doi:10.1016/j.jacr.2023.07.017
4. 4. Chhabra A, Gupta A, Thakur U, et al. Osseous TumorReporting and Data System-Multireader Validation Study. JComput Assist Tomogr. 2021;45(4):571-585. doi:10.1097/RCT.0000000000001184
5. 5. Chang CY, Garner HW, Ahlawat S, et al. Society of SkeletalRadiology- white paper. Guidelines for the diagnostic management of incidental solitary bone lesions on CT and MRI in adults: bone reporting and data system (Bone-RADS). Skeletal Radiol. 2022;51(9):1743-1764. doi:10.1007/S00256-022-04022-8
6. 6. Xing Y, Ding D, Dai S, et al. Bone reporting and data systemon CT (Bone-RADS-CT): a validation study by four readers on328 cases from three local and two public databases. Insights Imaging. 2025;16(1):174. doi:10.1186/S13244-025-02057-8
7. 7. Xing Y, Hu Y, Liu X, et al. Bone Reporting and Data Systemon MRI (Bone-RADS-MRI): a validation study by four readerson 275 cases from three local and two public databases. Insights Imaging. 2025;16(1). doi:10.1186/S13244-025-02040-3
8. 8. Doshi R, Amin KS, Khosla P, Bajaj S, Chheang S, Forman HP.Quantitative Evaluation of Large Language Models toStreamline Radiology Report Impressions: A Multimodal Retrospective Analysis. Radiology. 2024;310(3). doi: 10.1148/radiol.231593.

9. 9. Can E, Uller W, Vogt K, et al. Large Language Models forSimplified Interventional Radiology Reports: A Comparative Analysis. Acad Radiol. Published online September 30, 2024. doi:10.1016/J.ACRA.2024.09.041
10. 10. Bhayana R, Jajodia A, Chawla T, et al. Accuracy of LargeLanguage Model–based Automatic Calculation of Ovarian-Adnexal Reporting and Data System MRI Scores from PelvicMRI Reports. Radiology. 2025;315(1). doi: 10.1148/radiol.241554.
11. 11. Arnold PG, Russe MF, Bamberg F, et al. Performance of largelanguage models for CAD-RADS 2.0 classification derived from cardiac CT reports. J Cardiovasc Comput Tomogr. 2025;0(0). doi:10.1016/j.jcct.2025.03.007
12. 12. Güneş YC, Cesur T, Çamur E, Günbey Karabekmez L.Evaluating text and visual diagnostic capabilities of large language models on questions related to the Breast ImagingReporting and Data System Atlas 5th edition. Diagn IntervRadiol. 2025;31(2):111-129. doi:10.4274/DIR.2024.242876
13. 13. Cabezas E, Toro-Tobon D, Johnson T, et al. ChatGPT-4’sAccuracy in Estimating Thyroid Nodule Features and CancerRisk From Ultrasound Images. Endocrine Practice. 2025;31(6):716-723. doi:10.1016/j.eprac.2025.03.008
14. 14. Gisev N, Bell JS, Chen TF. Interrater agreement and interraterreliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy. 2013;9(3):330-338. doi:10.1016/j.sapharm.2012.04.004
15. 15. An JY, Unsdorfer KML, Weinreb JC. BI-RADS, C-RADS, CAD-RADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TI-RADS: Reporting and Data Systems.Radiographics. 2019;39(5):1435. doi:10.1148/RG.2019190087
16. 16. Park C, Azhideh A, Pooyan A, et al. Diagnostic performance and inter-reader reliability of bone reporting and data system (Bone-RADS) on computed tomography. Skeletal Radiol. 2025;54(2):209-217. doi:10.1007/S00256-024-04721-4
17. 17. Shen Y, Xu Y, Ma J, et al. Multi-modal large language modelsin radiology: principles, applications, and potential. Abdominal Radiology. 2025;50(6):2745-2757. doi:10.1007/S00261-024-04708-8/FIGURES/2
18. 18. Mukherjee P, Hou B, Suri A, et al. Evaluation of GPT largelanguage model performance on RSNA 2023 case of the day questions. Radiology. 2024;313(1). doi:10.1148/RADIOL.240609
19. 19. Suh PS, Shim WH, Suh CH, et al. Comparing Large LanguageModel and Human Reader Accuracy with New England Journal of Medicine Image Challenge Case Image. Radiology. 2024;313(3). doi:10.1148/RADIOL.241668
20. 20. Nakaura T, Yoshida N, Kobayashi N, et al. Performance ofMultimodal Large Language Models in Japanese Diagnostic Radiology Board Examinations (2021-2023). Acad Radiol. 2025;32(5):2394-2401. doi:10.1016/j.acra.2024.10.035
21. 21. Reith TP, D’Alessandro DM, D’Alessandro MP. Capability ofmultimodal large language models to interpret pediatric radiological images. Pediatr Radiol. 2024;54(10):1729-1737. doi:10.1007/S00247-024-06025-0
22. 22. Hou B, Mukherjee P, Batheja V, Wang KC, Summers RM, LuZ.One Year On: Assessing Progress of Multimodal LargeLanguage Model Performance on RSNA 2024 Case of the DayQuestions. Radiology. 2025;316(2). doi:10.1148/RADIOL.250617