Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination

Batuhan Ayhan; Samet Batuhan Yoğurt

Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination

Abstract

Aims: Large language models (LLMs) have demonstrated promising performance in medical examinations; however, their effectiveness in orthopedics-particularly in visually demanding questions-remains unclear. This study aimed to compare the performance of ChatGPT-5.2 and Gemini 3 Pro on orthopedics and traumatology questions from the Turkish Medical Specialization Examination (TUS). Methods: A total of 7.200 TUS questions from 2006 to 2021 were screened, and 163 orthopedics-related questions with validated answer keys were included. Questions were categorized into seven subspecialties and classified as text-based (n=154) or visual (n=9). Each question was independently presented to both models in Turkish without prior prompting. Responses were recorded as correct or incorrect. Statistical analyses included the McNemar test for paired comparisons and Fisher’s exact test for within-model comparisons of visual versus text-based performance. Results: ChatGPT-5.2 achieved an accuracy of 93.3% (152/163), while Gemini 3 Pro achieved 92.0% (150/163), with no statistically significant difference between models (p=0.593). Both models showed higher error rates on visual questions than on text-based questions (ChatGPT-5.2: 44.4% vs. 4.5%, p=0.001; Gemini 3 Pro: 33.3% vs. 6.5%, p=0.025); however, these visual comparisons were based on only nine items and should be regarded as exploratory. Error distribution across subspecialties was similar between models, with orthopedic trauma showing the highest error frequency. Conclusion: Both ChatGPT-5.2 and Gemini 3 Pro demonstrated high overall accuracy in orthopedic examination questions, with no significant difference between models. However, both models exhibited a marked decline in performance on visually based questions, highlighting persistent limitations in multimodal reasoning. Future research should incorporate larger visual datasets to better evaluate AI performance in orthopedic contexts.

Keywords

References

Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi:10.1038/s41586-023-06291-2
Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the United States Medical Licensing Examination? JMIR Med Educ. 2023;9:e45312. doi:10.2196/45312
Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198. doi:10.1371/journal.pdig.0000198
Thirunavukarasu AJ, Ting DSJ, Elangovan K, et al. Large language models in medicine. Nat Med. 2023;29(8):1930-1940. doi:10.1038/s41591-023-02448-8
Ayers JW, Poliak A, Dredze M, et al. Comparing physician and Artificial Intelligence chatbot responses to patient questions. JAMA Intern Med. 2023;183(6):589-596. doi:10.1001/jamainternmed.2023.1838
Wang H, Wu W, Dou Z, He L, Yang L. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inform. 2023;177:105173. doi:10.1016/j.ijmedinf.2023.105173
Liu M, Okuhara T, Chang X, et al. Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis. J Med Internet Res. 2024;26:e60807. doi:10.2196/60807
Oztermeli AD, Oztermeli A. ChatGPT performance in the medical specialty exam: an observational study. Medicine (Baltimore). 2023;102(32):e34673. doi:10.1097/MD.0000000000034673

OpenAI. GPT-4 technical report. Published 2023. Accessed March 24, 2026. https://arxiv.org/abs/2303.08774
Keskin A, Aygün T. Comparison of the success of ChatGPT 4.0 and Google Gemini in anatomy questions asked in Turkiye national medical specialization exams. Tıp Eğitimi Dunyası. 2025;24:127-134. doi:10.25282/ted.1716591
Jung LB, Gudera JA, Wiegand TLT, Allmendinger S, Dimitriadis K, Koerte IK. ChatGPT passes German state examination in medicine with picture questions omitted. Dtsch Arztebl Int. 2023;120(21):373-374. doi:10.3238/arztebl.m2023.0113
Ali R, Tang OY, Connolly ID, et al. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery. 2023;93(5):1090-1098. doi:10.1227/neu.0000000000002551
Alessandri Bonetti M, Giorgino R, Gallo Afflitto G, De Lorenzi F, Egro FM. How does ChatGPT perform on the Italian residency admission national exam compared to 15,869 medical graduates? Ann Biomed Eng. 2024;52(4):745-749. doi:10.1007/s10439-023-03318-7
Lai UH, Wu KS, Hsu TY, et al. Evaluating ChatGPT-4 on the UK Medical Licensing Assessment. Front Med (Lausanne). 2023;10:1240915. doi:10.3389/fmed.2023.1240915
Taloni A, Borselli M, Scarsi V, et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep. 2023;13(1):18562. doi:10.1038/s41598-023-45837-2
Hayden N, Gilbert S, Poisson LM, et al. Performance of GPT-4 with vision on text-and image-based ACR diagnostic radiology in-training examination questions. Radiology. 2024;312(3):e240153. doi:10.1148/radiol.240153
Nicikowski J, Szczepański M, Miedziaszczyk M, Kudliński B. The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland. Clin Kidney J. 2024;17(8):sfae193. doi:10.1093/ckj/sfae193
Lum ZC. Can artificial intelligence pass the American Board of Orthopaedic Surgery Examination? Orthopaedic residents versus ChatGPT. Clin Orthop Relat Res. 2023;481(8):1623-1630. doi:10.1097/CORR.0000000000002704
Kung JE, Marshall C, Gauthier C, Gonzalez TA, Jackson JB 3rd. Evaluating ChatGPT performance on the orthopaedic in-training examination. JB JS Open Access. 2023;8(3):e23.00056. doi:10.2106/JBJS.OA.23.00056
Sparks CA, Kraeutler MJ, Chester GA, et al. Inadequate performance of ChatGPT on orthopedic board-style written exams. Cureus. 2024;16(6):e62643. doi:10.7759/cureus.62643
Hofmann HL, Guerra GA, Le JL, et al. The rapid development of artificial intelligence: GPT-4’s performance on orthopedic surgery board questions. Orthopedics. 2024;47(2):e85-e89. doi:10.3928/01477447-20230922-05
Yiğitbay A. Evaluation of ChatGPT’s performance in the Turkish Board of Orthopaedic Surgery Examination. Med Bull Haseki. 2024;62(4):243-249. doi:10.4274/haseki.galenos.2024.10038
Yağar H, Gümüşoğlu E, Asfuroğlu ZM. Assessing the performance of ChatGPT-4o on the Turkish Orthopedics and Traumatology Board Examination. Jt Dis Relat Surg. 2025;36(2):304-310. doi:10.52312/jdrs.2025.1958
İpek E, Sulek Y, Balkanlı B. Performance of AI models vs orthopedic residents in Turkish Specialty Training Development Exams in orthopedics. Med Bull Sisli Etfal Hosp. 2025;59(2):151-155. doi:10.14744/SEMB.2025.65289
Sismanoglu S, Capan BS. Performance of Artificial Intelligence on Turkish dental specialization exam: can ChatGPT-4.0 and Gemini Advanced achieve comparable results to humans? BMC Med Educ. 2025; 25(1):214. doi:10.1186/s12909-024-06389-9
Yay Kuşçu HY, Görüş Z. Performance of large language models on prosthodontics questions of the dentistry specialization examination: a comparative analysis (2014-2024). Anatolian Curr Med J. 2025;7(6):893-899. doi:10.38053/acmj.1789931
Mutlu H, Kokulu K, Sert ET, Topuz MA. Evaluation of ChatGPT’s performance in Türkiye’s first emergency medicine sub-specialization exam. Eurasian J Emerg Med. 2025;24(3):171-175. doi:10.4274/eajem.galenos.2025.32644
Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: comparison study. JMIR Med Educ. 2023;9:e48002. doi:10.2196/48002

Details

Primary Language

English

Subjects

Orthopaedics

Journal Section

Research Article

Authors

Batuhan Ayhan ^*
0000-0001-5848-1618
Türkiye

Samet Batuhan Yoğurt
0000-0002-0703-046X
Türkiye

Publication Date

July 16, 2026

Submission Date

April 1, 2026

Acceptance Date

July 2, 2026

Published in Issue

Year 2026 Volume: 8 Number: 4

IZ

https://izlik.org/JA83CH35RM

Cite

RIS / Bibtex

APA

Ayhan, B., & Yoğurt, S. B. (2026). Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination. Anatolian Current Medical Journal, 8(4), 747-751. https://izlik.org/JA83CH35RM

AMA

1.Ayhan B, Yoğurt SB. Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination. Anatolian Curr Med J / ACMJ / acmj. 2026;8(4):747-751. https://izlik.org/JA83CH35RM

Chicago

Ayhan, Batuhan, and Samet Batuhan Yoğurt. 2026. “Comparative Performance of ChatGPT-5.2 and Gemini 3 Pro in Orthopedics Questions of the Medical Specialization Examination”. Anatolian Current Medical Journal 8 (4): 747-51. https://izlik.org/JA83CH35RM.

EndNote

Ayhan B, Yoğurt SB (July 1, 2026) Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination. Anatolian Current Medical Journal 8 4 747–751.

IEEE

[1]B. Ayhan and S. B. Yoğurt, “Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination”, Anatolian Curr Med J / ACMJ / acmj, vol. 8, no. 4, pp. 747–751, July 2026, [Online]. Available: https://izlik.org/JA83CH35RM

ISNAD

Ayhan, Batuhan - Yoğurt, Samet Batuhan. “Comparative Performance of ChatGPT-5.2 and Gemini 3 Pro in Orthopedics Questions of the Medical Specialization Examination”. Anatolian Current Medical Journal 8/4 (July 1, 2026): 747-751. https://izlik.org/JA83CH35RM.

JAMA

1.Ayhan B, Yoğurt SB. Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination. Anatolian Curr Med J / ACMJ / acmj. 2026;8:747–751.

MLA

Ayhan, Batuhan, and Samet Batuhan Yoğurt. “Comparative Performance of ChatGPT-5.2 and Gemini 3 Pro in Orthopedics Questions of the Medical Specialization Examination”. Anatolian Current Medical Journal, vol. 8, no. 4, July 2026, pp. 747-51, https://izlik.org/JA83CH35RM.

Vancouver

1.Batuhan Ayhan, Samet Batuhan Yoğurt. Comparative performance of ChatGPT-5.2 and Gemini 3 Pro in orthopedics questions of the medical specialization examination. Anatolian Curr Med J / ACMJ / acmj [Internet]. 2026 Jul. 1;8(4):747-51. Available from: https://izlik.org/JA83CH35RM