Cross-Linguistic Evaluation of Artificial Intelligence Chatbots: Performance of ChatGPT-3.5, Copilot and Gemini in Neuro-ophthalmologic Evaluation in English and Turkish
Öz
Abstract
Background/Aims: To evaluate the performance of ChatGPT-3.5, Copilot, and Gemini artificial intelligence chatbots on the same questions in neuro-ophthalmologic evaluation in English and Turkish.
Methods: Forty questions related to neuro-ophthalmology were included in the study. After all English questions were translated into Turkish by a certified native speaker, both versions of the questions were asked to ChatGPT-3.5, Copilot, and Gemini chatbots. The answers were compared with the answer key and grouped as correct and incorrect. Their superiority over each other was compared statistically.
Results: ChatGPT-3,5 47.5%, Copilot 57.5%, and Gemini 32.5% answered the English questions correctly. ChatGPT-3,5 57.5%, Copilot 52.5%, and Gemini 32.5% answered the questions correctly in Turkish. No statistically significant difference was detected between chatbots in answering the same questions in English and Turkish, although there were different levels of success (p>0.05).
Conclusions: Although there is no statistically significant difference, chatbots can answer the same questions differently. In addition to improving the knowledge level of chatbots, their language skills also need to be improved.
Anahtar Kelimeler
ChatGPT-3.5, Copilot, Gemini, Neuro-ophthalmology, Turkish, Artificial intelligence applications
Etik Beyan
Since the data in our study is not from any animal or human sources, ethics committee approval is not required.
Kaynakça
- 1. Madadi Y, Delsoz M, Lao PA, Fong JW, Hollingsworth T, Kahook MY, et al. ChatGPT Assisting Diagnosis of Neuro-ophthalmology Diseases Based on Case Reports. medRxiv 2023.
- 2. Stunkel L, Sharma RA, Mackay DD, Wilson B, Van Stavern GP, Newman NJ, et al. Patient Harm Due to Diagnostic Error of Neuro-Ophthalmologic Conditions. Ophthalmology 2021; 128:1356–1362.
- 3. Frohman LP. The human resource crisis in neuro-ophthalmology. J Neuroophthalmol 2008; 28:231–234.
- 4. Debusk A, Subramanian PS, Scannell Bryan M, Moster ML, Calvert PC, and Frohman LP. Mismatch in Supply and Demand for Neuro-Ophthalmic Care. J Neuroophthalmol 2022; 42:62–67.
- 5. Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol 2019; 103:167.
- 6. Sensoy E and Citirik M. A comparative study on the knowledge levels of artificial intelligence programs in diagnosing ophthalmic pathologies and intraocular tumors evaluated their superiority and potential utility. Int Ophthalmol 2023; 43:4905–4909.
- 7. Bhatti TM, Chen JJ, Danesh-Meyer H V., Levin LA, Moss HE, Philips PH, et al., editors. Neuro-Ophthalmology. San Francisco: American Academy of Ophthalmology; 2023.
- 8. Şensoy E, Çıtırık M. ChatGPT-3.5, Copilot ve Gemini'nin oküler inflamasyon ve üveit konusundaki çoktan seçmeli sorularda performans analizi: Dil farklılıklarının etkisi: Kesitsel araştırma. Turkiye Klinikleri J Ophthalmol 2025;34:12-16
- 9. Şensoy E and Çıtırık M. Performance of chatgptChatGPT-3.5, copilot Copilot, and gemini Gemini in answering english English and turkish Turkish questions related to ocular surface diseases and cornea: a comparison study. Turkish Journal of Clinical and Experimental Ophthalmology 2025; 20:37-41.
- 10. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS digital health 2023; 2:e0000198.