Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist

Osman Konukoğlu; Murat Kaya; Şeyma Eseoğlu Pekcan; İsa Günaydin

doi:10.65092/autfm.1878887

TR EN

Omuz Kırıklarında ChatGPT’nin Tanısal Doğruluğu: Radyoloji ve Acil Tıp Uzmanı ile Karşılaştırmalı Bir Çalışma

Abstract

Arka plan Chat Generative Pre-Trained Transformer (ChatGPT), gerçek yaşam sohbetlerini simüle edebilen ve tanısal destek sağlayabilen yeni bir büyük dil modelidir. Ancak omuz travmalarında kırık saptamadaki performansı henüz araştırılmamıştır. Amaç Bu çalışmanın amacı, ChatGPT’nin omuz kırıklarını saptamadaki tanısal doğruluğunu değerlendirmek ve performansını bir acil tıp uzmanı ile bir radyologla karşılaştırmaktır. Yöntemler Bu retrospektif çalışmaya, Eylül 2023 ile Temmuz 2025 tarihleri arasında hem omuz grafisi hem de bilgisayarlı tomografi (BT) çekilen 197 hasta dahil edildi. Bir acil tıp uzmanı, bir radyolog ve ChatGPT modelleri (4o ve 4.5), anonimleştirilmiş grafileri bağımsız ve kör olarak değerlendirerek kırık varlığını ve lokalizasyonunu belirledi. BT altın standart olarak kabul edildi. Tanısal performans; duyarlılık, özgüllük, doğruluk ve eğri altındaki alan (AUC) kullanılarak değerlendirildi. Bulgular 197 hastanın 74’ünde (%37,56) kırık saptandı ve en sık humerus tutulumu izlendi. Radyolog en yüksek tanısal performansı gösterdi (AUC: 0,903; Doğruluk: %90,86), bunu acil tıp uzmanı izledi (AUC: 0,824; Doğruluk: %82,74). ChatGPT-4o (AUC: 0,641; Doğruluk: %61,93) ve ChatGPT-4.5 (AUC: 0,626; Doğruluk: %57,36) istatistiksel olarak anlamlı ancak zayıf-orta düzeyde tanısal katkı sağladı; her iki model de yüksek duyarlılık ancak düşük özgüllük gösterdi. Her iki modelde de gözlemciler arası uyum stabil bulundu. Sonuç ChatGPT modelleri, omuz kırıklarının tanısında klinisyenlere kıyasla sınırlı performans göstermiştir. Klinik uzmanlığın yerini alamamakla birlikte, karar verme süreçlerinde destekleyici araçlar olarak kullanılabilirler.

Keywords

Omuz travması, Kırık, Radyografi, ChatGPT, Radyoloji

Ethical Statement

Gaziantep Şehir Hastanesi Girişimsel Olmayan Klinik Araştırmalar Yerel Etik Kurulu Koordinatörlüğü’nden, 18/06/2025 tarihinde gerçekleştirilen toplantıda 212/2025 karar numarası ile onay alınmıştır.

Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist

Abstract

Background Chat Generative Pre-Trained Transformer (ChatGPT) is a new large language model capable of simulating real-life conversations and providing diagnostic support. However, its performance in detecting fractures in shoulder traumas has not been investigated. Objective To evaluate the diagnostic accuracy of ChatGPT in detecting shoulder fractures and to compare its performance with an emergency medicine specialist and a radiologist. Methods This retrospective study included 197 patients who underwent both shoulder radiography and computed tomography (CT) between September 2023 and July 2025. One emergency medicine specialist, one radiologist, and ChatGPT models (4o and 4.5) independently and blindly evaluated anonymized radiographs to determine the presence and location of fractures. CT was accepted as the gold standard. Diagnostic performance was assessed using sensitivity, specificity, accuracy, and area under the curve (AUC) values. Results Among 197 patients, 74 (37.56%) had fractures, most commonly involving the humerus. The radiologist demonstrated the highest diagnostic performance (AUC: 0.903; Accuracy: 90.86%), followed by the emergency physician (AUC: 0.824; Accuracy: 82.74%). ChatGPT-4o (AUC: 0.641; Accuracy: 61.93%) and ChatGPT-4.5 (AUC: 0.626; Accuracy: 57.36%) showed statistically significant but weak-to-moderate diagnostic contribution, with high sensitivity but poor specificity. Both models demonstrated stable intra-rater agreement. Conclusion ChatGPT models showed limited performance compared with clinicians in diagnosing shoulder fractures. While they cannot replace clinical expertise, they may serve as supportive tools in decision-making.

Keywords

Shoulder trauma, Fracture, X-Ray, ChatGPT, Radiology

Ethical Statement

Approval from the Gaziantep City Hospital Coordinatorship of Local Ethics Comittee for Non-Interventional Clinical Researches was obtained during the meeting held on 18/06/2025, under decision number 212/2025.

References

Baker HP, Dwyer E, Kalidoss S, et al. ChatGPT's Ability to Assist with Clinical Documentation: A Randomized Controlled Trial. J Am Acad Orthop Surg. 2024;32(3):123-129. https://doi.org/10.5435/jaaos-d-23-00474
Chalhoub R, Mouawad A, Aoun M, et al. Will ChatGPT be Able to Replace a Spine Surgeon in the Clinical Setting? World Neurosurg. 2024;185:e648-e652. https://doi.org/10.1016/j.wneu.2024.02.101
Liu S, Wright AP, Patterson BL, et al. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc. 2023;30(7):1237-1245. https://doi.org/10.1093/jamia/ocad072
Casey JC, Dworkin M, Winschel J, et al. ChatGPT: A concise Google alternative for people seeking accurate and comprehensive carpal tunnel syndrome information. Hand Surg Rehabil. 2024;43(5):101757. https://doi.org/10.1016/j.hansur.2024.101757
Horiuchi D, Tatekawa H, Oura T, et al. ChatGPT's diagnostic performance based on textual vs. visual information compared to radiologists' diagnostic performance in musculoskeletal radiology. Eur Radiol. 2025;35(1):506-516. https://doi.org/10.1007/s00330-024-10902-5
Horiuchi D, Tatekawa H, Shimono T, et al. Accuracy of ChatGPT generated diagnosis from patient's medical history and imaging findings in neuroradiology cases. Neuroradiology. 2024;66(1):73-79. https://doi.org/10.1007/s00234-023-03252-4
De Angelis L, Baglivo F, Arzilli G, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1166120. https://doi.org/10.3389/fpubh.2023.1166120
Kachman MM, Brennan I, Oskvarek JJ, et al. How artificial intelligence could transform emergency care. Am J Emerg Med. 2024;81:40-46. https://doi.org/10.1016/j.ajem.2024.04.024
Meral G, Ates S, Gunay S, et al. Comparative analysis of ChatGPT, Gemini and emergency medicine specialist in ESI triage assessment. Am J Emerg Med. 2024;81:146-150. https://doi.org/10.1016/j.ajem.2024.05.001
Pasli S, Sahin AS, Beser MF, et al. Assessing the precision of artificial intelligence in ED triage decisions: Insights from a study with ChatGPT. Am J Emerg Med. 2024;78:170-175. https://doi.org/10.1016/j.ajem.2024.01.037

Cha Y, Kim JT, Park CH, et al. Artificial intelligence and machine learning on diagnosis and classification of hip fracture: systematic review. J Orthop Surg Res. 2022;17(1):520. https://doi.org/10.1186/s13018-022-03408-7
Hoppe JM, Auer MK, Struven A, et al. ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis. J Med Internet Res. 2024;26:e56110. https://doi.org/10.2196/56110
Mohammadi M, Parviz S, Parvaz P, et al. Diagnostic performance of ChatGPT in tibial plateau fracture in knee X-ray. Emerg Radiol. 2025;32(1):59-64. https://doi.org/10.1007/s10140-024-02298-y
Ridpath CA, Wilson AJ. Shoulder and humerus trauma. Semin Musculoskelet Radiol. 2000;4(2):151-170. https://doi.org/10.1055/s-2000-13010
Mese I, Taslicay CA, Sivrioglu AK. Improving radiology workflow using ChatGPT and artificial intelligence. Clin Imaging. 2023;103:109993. https://doi.org/10.1016/j.clinimag.2023.109993
Olczak J, Fahlberg N, Maki A, et al. Artificial intelligence for analyzing orthopedic trauma radiographs. Acta Orthop. 2017;88(6):581-586. https://doi.org/10.1080/17453674.2017.1344459
Kuo RYL, Harrison C, Curran TA, et al. Artificial Intelligence in Fracture Detection: A Systematic Review and Meta-Analysis. Radiology. 2022;304(1):50-62. https://doi.org/10.1148/radiol.211785
Bousson V, Attane G, Benoist N, et al. Artificial Intelligence for Detecting Acute Fractures in Patients Admitted to an Emergency Department: Real-Life Performance of Three Commercial Algorithms. Acad Radiol. 2023;30(10):2118-2139. https://doi.org/10.1016/j.acra.2023.06.016
Hiredesai AN, Martinez CJ, Anderson ML, et al. Is Artificial Intelligence the Future of Radiology? Accuracy of ChatGPT in Radiologic Diagnosis of Upper Extremity Bony Pathology. Hand (N Y). 2024:15589447241298982. https://doi.org/10.1177/15589447241298982
Bradley AT, Adler JA, Curtis DM, et al. Are Oblique Views Necessary? A Review of the Clinical Value of Oblique Knee Radiographs in the Acute Setting. West J Emerg Med. 2022;23(6):939-946. https://doi.org/10.5811/westjem.2022.8.56453
Gültekin A, Gök Ü, Uyar A, et al. Diagnostic performance of ChatGPT-4.0 in elbow fracture detection: A comparative study of radial head, distal humerus, and olecranon fractures. Medicine (Baltimore). 2025;104(37):e43716. https://doi.org/10.1097/md.0000000000043716
Kirilmaz A, Erdem TE, Yaka H, et al. Diagnosis of shoulder dislocation on AP radiographs: A comparative analysis of diagnostic performance between orthopedic surgeons, emergency physicians, and ChatGPT models. Injury. 2026;57(2):112957. https://doi.org/10.1016/j.injury.2025.112957

Details

Primary Language

English

Subjects

Radiology and Organ Imaging

Journal Section

Research Article

Authors

Osman Konukoğlu ^*
0000-0002-3353-0230
Türkiye

Murat Kaya
0000-0002-1251-1292
Türkiye

Şeyma Eseoğlu Pekcan
0009-0000-0885-3532
Türkiye

İsa Günaydin
0009-0002-9503-2517
Türkiye

Publication Date

June 30, 2026

Submission Date

February 1, 2026

Acceptance Date

June 10, 2026

Published in Issue

Year 2026 Volume: 79 Number: 2

DOI

https://doi.org/10.65092/autfm.1878887

IZ

https://izlik.org/JA27TZ43TW

APA

Konukoğlu, O., Kaya, M., Eseoğlu Pekcan, Ş., & Günaydin, İ. (2026). Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist. Ankara Üniversitesi Tıp Fakültesi Mecmuası, 79(2), 216-225. https://doi.org/10.65092/autfm.1878887

AMA

1.Konukoğlu O, Kaya M, Eseoğlu Pekcan Ş, Günaydin İ. Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist. Ankara Üniversitesi Tıp Fakültesi Mecmuası. 2026;79(2):216-225. doi:10.65092/autfm.1878887

Chicago

Konukoğlu, Osman, Murat Kaya, Şeyma Eseoğlu Pekcan, and İsa Günaydin. 2026. “Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study With an Emergency Medicine Specialist and a Radiologist”. Ankara Üniversitesi Tıp Fakültesi Mecmuası 79 (2): 216-25. https://doi.org/10.65092/autfm.1878887.

EndNote

Konukoğlu O, Kaya M, Eseoğlu Pekcan Ş, Günaydin İ (June 1, 2026) Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist. Ankara Üniversitesi Tıp Fakültesi Mecmuası 79 2 216–225.

IEEE

[1]O. Konukoğlu, M. Kaya, Ş. Eseoğlu Pekcan, and İ. Günaydin, “Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist”, Ankara Üniversitesi Tıp Fakültesi Mecmuası, vol. 79, no. 2, pp. 216–225, June 2026, doi: 10.65092/autfm.1878887.

ISNAD

Konukoğlu, Osman - Kaya, Murat - Eseoğlu Pekcan, Şeyma - Günaydin, İsa. “Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study With an Emergency Medicine Specialist and a Radiologist”. Ankara Üniversitesi Tıp Fakültesi Mecmuası 79/2 (June 1, 2026): 216-225. https://doi.org/10.65092/autfm.1878887.

JAMA

1.Konukoğlu O, Kaya M, Eseoğlu Pekcan Ş, Günaydin İ. Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist. Ankara Üniversitesi Tıp Fakültesi Mecmuası. 2026;79:216–225.

MLA

Konukoğlu, Osman, et al. “Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study With an Emergency Medicine Specialist and a Radiologist”. Ankara Üniversitesi Tıp Fakültesi Mecmuası, vol. 79, no. 2, June 2026, pp. 216-25, doi:10.65092/autfm.1878887.

Vancouver

1.Osman Konukoğlu, Murat Kaya, Şeyma Eseoğlu Pekcan, İsa Günaydin. Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist. Ankara Üniversitesi Tıp Fakültesi Mecmuası. 2026 Jun. 1;79(2):216-25. doi:10.65092/autfm.1878887

Omuz Kırıklarında ChatGPT’nin Tanısal Doğruluğu: Radyoloji ve Acil Tıp Uzmanı ile Karşılaştırmalı Bir Çalışma

Abstract

Keywords

Ethical Statement

Diagnostic Accuracy of ChatGPT in Shoulder Fractures: A Comparative Study with an Emergency Medicine Specialist and a Radiologist

Abstract

Keywords

Ethical Statement

References

Details

Primary Language

Subjects

Journal Section

Authors

Publication Date

Submission Date

Acceptance Date

Published in Issue

DOI

IZ

Cite