Performance of Large Language Models in Ophthalmology Questions: Do Language Differences Matter?

Öz

Aim: To evaluate the effectiveness of ChatGPT-4.0, Copilot AI, Gemini AI, and Claude AI chatbots in answering the fundamental principles of ophthalmology. 

Materials and Methods: All forty questions in the study questions section of the American Academy of Ophthalmology (AAO) 2024-2025 Basic and Clinical Science Course (BCSC) Fundamentals and Principles of Ophthalmology book were asked to ChatGPT-4.0, Copilot AI, Gemini AI, and Claude AI in both English and Turkish. The questions were asked only once and at different times of the day for each language. The responses provided by the chatbots were compared with the official answer key provided at the end of the book and classified as correct or incorrect. 

Results: For English questions, Copilot AI (95%) and Claude AI (92.5%) showed higher accuracy than ChatGPT-4.0 (75%) and Gemini AI (87.5%). In Turkish questions, the accuracy rates of all models were found to be close to each other (85–92.5%), and no significant superiority was detected. Across languages, all AI applications showed medium-high consistency. The highest level of consistency was detected in Copilot AI and Claude AI. 

Conclusion: Although Copilot AI and Claude AI outperformed the other two bots in English questions, they did not demonstrate a meaningful superiority over each other in Turkish questions.

Anahtar Kelimeler

• ChatGPT- 4.0
• Copilot
• Gemini
• Claude
• Ophthalmology Education

Destekleyen Kurum

Not applicable.

Etik Beyan

The evaluatıon of chatbot responces was conducted by a single experienced ophthalmologist. Since our study did not involve human or animal data, no ethics committee approval was obtained.

Kaynakça

1. Ozsoy S, Aydin D, Ekici F. Effects of modafinil on pentylenetetrazol-induced convulsive epilepsy. Bratisl Lek Listy 2015; 116 (3): 162-166.
2. Swapnajeet S, Subodh B, Gourav G. Modafinil dependence and hypersexuality: a case report and review of the evidence. Clin Psychopharmacol Neurosci. 2016; 14: 402-404.
3. Ballon JS, Feifel D. A systematic review of modafinil: potential clinical uses and mechanisms of action. J Clin Psychiatry 2006; 67: 554-566.
4. Schwartz JR. Modafinil: new indications for wake promotion. Expert Opin Pharmacother 2005; 6: 115-129.
5. Wu P, Jones S, Ryan CJ, Michail D, Robinson TD. Modafinil-induced psychosis. Intern Med J. 2008; 38: 677-678.
6. Mariani JJ, Hart CL. Psychosis associated with modafinil and shift work. Am J Psychiatry. 2005; 162: 1983.
7. Özer S, Demir B. Hypomania in a Schizophrenic Patient Treated with Modafinil for Clozapine-Induced Sedation. Arch Neuropsychiatr. 2010; 47: 171-173.
8. Aggarwal A, Garg A, Jiloha RC. Psychotic relapse in a patient with schizophrenia associated with modafinil therapy. Primary Psychiatry. 2009; 16: 26-27.

9. Narendran R, Young CM, Valenti AM, Nickolova MK, Pristach CA. Is psychosis exacerbated by modafinil? Arch Gen Psychiatry. 2002; 59: 292-293.
10. McClellan KJ, Spencer CM. Modafinil: A review of its pharmacology and clinical efficacy in the management of narcolepsy. CNS Drugs. 1998; 9(4): 311-324.
11. Wolf J, Fiedler U, Anghelescu I, Schwertfeger N. Manic switch in a patient with treatment-resistant bipolar depression treated with modafinil. Journal of Clinical Psychiatry. 2006; 67(11): 1817.