Ortopedik travma karar verme sürecinde ChatGPT, Google Gemini ve ortopedi cerrahlarının karşılaştırmalı değerlendirmesi

Öz

Arka Plan: Büyük dil modelleri (LLM’ler), tıp eğitimi ve klinik destek dahil olmak üzere sağlık hizmetlerinin birçok alanında giderek daha fazla kullanılmaktadır. Bununla birlikte, ortopedik travma gibi yüksek riskli klinik alanlardaki güvenilirlikleri henüz net değildir. Amaç: Bu çalışma, ortopedik travma sorularında ChatGPT ve Google Gemini’nin doğruluk, klinik muhakeme performansı ve potansiyel klinik risk düzeylerini karşılaştırmayı ve performanslarını ortopedi cerrahları ile kıyaslamayı amaçlamıştır. Yöntemler: Üst ekstremite, alt ekstremite, pediatrik travma ve acil durumları kapsayan toplam 60 ortopedik travma sorusu (30 teorik ve 30 klinik senaryo temelli) hazırlanmıştır. Yanıtlar ChatGPT, Google Gemini ve iki bağımsız ortopedi cerrahından elde edilmiştir. Tüm yanıtlar anonimleştirilmiş ve iki kıdemli ortopedi uzmanı tarafından doğru, kısmen doğru, yanlış veya potansiyel olarak klinik açıdan güvensiz olarak değerlendirilmiştir. İstatistiksel analizler ki-kare ve Fisher’ın kesin testi kullanılarak yapılmıştır. Bulgular: Genel doğruluk oranı ChatGPT için %83,3, Gemini için %80,0 ve ortopedi cerrahları için %91,7 olarak bulunmuştur (p < 0,05). Yapay zekâ sistemleri teorik sorularda klinik senaryolara kıyasla daha iyi performans göstermiştir. Potansiyel olarak klinik açıdan güvensiz yanıtlar ChatGPT yanıtlarının %5,0’ında, Gemini yanıtlarının %8,3’ünde ve cerrah yanıtlarının %1,7’sinde görülmüştür. Güvensiz yanıtlar açısından cerrahlar ile Gemini arasındaki fark istatistiksel olarak anlamlı bulunmuştur. Sonuç: Güncel büyük dil modelleri ortopedik travma sorularında yüksek doğruluk göstermesine rağmen, özellikle klinik senaryo temelli muhakemede eğitimli ortopedi cerrahlarının gerisinde kalmaktadır. LLM’ler destekleyici eğitim araçları olarak hizmet edebilir; ancak yüksek riskli travma karar verme süreçlerinde insan denetimi zorunlu olmaya devam etmektedir.

Anahtar Kelimeler

• yapay zeka
• büyük dil modelleri
• ortopedik travma
• klinik yetkinlik
• hasta güvenliği

Comparative evaluation of ChatGPT, Google Gemini and orthopedic surgeons in orthopedic trauma decision-making

Öz

Background: Large language models (LLMs) are increasingly used in medical education and clinical support. However, their reliability in high-risk clinical fields such as orthopedic trauma remains unclear. Purpose: This study aimed to compare the accuracy, clinical reasoning performance, and potential clinical risk of ChatGPT and Google Gemini in orthopedic trauma questions, and to benchmark their performance against orthopedic surgeons. Methods: A total of 60 orthopedic trauma questions (30 theoretical and 30 clinical scenario–based) were developed across upper extremity, lower extremity, pediatric trauma, and emergency conditions. Responses were obtained from ChatGPT, Google Gemini, and two independent orthopedic surgeons. All answers were anonymized and evaluated by two senior orthopedic specialists as correct, partially correct, incorrect, or potentially clinically unsafe. Statistical analyses were performed using chi-square and Fisher’s exact tests. Results: Overall accuracy was 83.3% for ChatGPT, 80.0% for Gemini, and 91.7% for orthopedic surgeons (p < 0.05). AI systems performed better in theoretical questions than in clinical scenarios. Potentially clinically unsafe responses occurred in 5.0% of ChatGPT answers, 8.3% of Gemini answers, and 1.7% of surgeon responses. The difference in unsafe responses between surgeons and Gemini was statistically significant. Conclusion: Although contemporary LLMs demonstrate high accuracy in orthopedic trauma questions, they remain inferior to trained orthopedic surgeons, particularly in clinical scenario–based reasoning. LLMs may serve as supportive educational tools, but human oversight remains essential in high-stakes trauma decision-making.

Anahtar Kelimeler

• Artificial Intelligence
• Large Language Models
• Orthopedic Trauma
• Clinical Competence
• Patient Safety.

Destekleyen Kurum

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Etik Beyan

This study did not involve human participants, identifiable patient data, biological samples, or clinical interventions. All materials consisted of standardized academic questions and fictional clinical scenarios developed solely for research purposes. No real patient information was used at any stage of the study. The participating orthopedic surgeons voluntarily provided responses to academic questions in a non-clinical setting. According to local institutional policies and national research regulations, studies that do not involve human subjects or patient data do not require formal ethics committee or institutional review board approval. Therefore, ethical approval was not required for this study.

Teşekkür

The authors would like to thank Cem Yıldırım and Osman Gorkem Muratoglu for their valuable assistance in the preparation and development of the orthopedic trauma questions used in this study.

Kaynakça

1. Thirunavukarasu AJ, Ting DSJ, Elangovan K, et al. Large language models in medicine. BMJ. 2023;381:p1269.
2. Van Dis EAM, Bollen J, Zuidema W, et al. ChatGPT: Five priorities for research. Nature. 2023;614(7947):224-6.
3. Sallam M. The utility of ChatGPT in healthcare education, research, and practice: A systematic review. Healthcare (Basel). 2023;11(6):887.
4. Rajpurkar P, Chen E, Banerjee O, et al. AI in health and medicine. Nat Med. 2022;28(1):31-8.
5. Court-Brown CM, Caesar B. Epidemiology of adult fractures:A review. Injury. 2006;37(8):691-7.
6. Giannoudis PV, Harwood PJ, Kontakis G, et al. Long-term quality of life in trauma patients following the full spectrum of tibial injury (fasciotomy, closed fracture, grade IIIB/IIIC open fracture and amputation). Injury. 2009;40(2):213-9.
7. Cascella M, Montomoli J, Bellini V, et al. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J Med Syst. 2023;47(1):33.
8. Court-Brown CM, Heckman JD, McQueen MM, et al., editors. Rockwood and Green’s fractures in adults. 9th ed. Philadelphia: Wolters Kluwer; 2020.

9. American Academy of Orthopaedic Surgeons. Management of hip fractures in older adults: evidence-based clinical practice guideline. Rosemont: American Academy of Orthopaedic Surgeons; 2014.
10. Meinberg EG, Agel J, Roberts CS, et al., editors. AO principles of fracture management. 3rd ed. Stuttgart: Thieme; 2018.
11. Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on the United States Medical Licensing Examination: potential for AI-assisted medical education. PLoS Digit Health. 2023;2(2):e0000198.
12. Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the United States Medical Licensing Examination? JMIR Med Educ. 2023;9:e45312.
13. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620:172-80.
14. Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence chatbot responses to patient questions. JAMA Intern Med. 2023;183(6):589-96.
15. Kelly CJ, Karthikesalingam A, Suleyman M, et al. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 2019;17:195.