Assessing the Impact of Language Differences on Artificial Intelligence Chatbot Performance in Ophthalmic Pathologies and Intraocular Tumors: A Study of ChatGPT-3.5, Copilot, and Gemini

Öz

Background: To investigate the effect of language differences on the success of ChatGPT-3.5, Copilot, and Gemini artificial intelligence chatbots in multiple-choice questions related to ophthalmic pathologies and intraocular tumors. Materials and Methods: Thirty-six English questions testing knowledge about ophthalmic pathologies and intraocular tumors were included in the study. These questions were asked to ChatGPT-3.5, Copilot, and Gemini chatbots in both English and Turkish after the Turkish translations were realized by a certified translator (native speaker). The answers given were compared with the answer key and grouped as correct and incorrect. Results: ChatGPT-3.5, Copilot, and Gemini answered the questions in English correctly at a rate of 75%, 66.7%, and 63.9%, respectively. These programs answered the Turkish questions correctly at a rate of 63.9%, 66.7%, and 69.4%, respectively. Although there were different rates of correct answers between chatbots in answering the Turkish versions of the questions, no statistically significant difference was detected (p>0.05). Conclusions: In addition to improving the knowledge of artificial intelligence chatbots, their ability to understand different languages, translate, and generate ideas needs to be improved to create the same perception in different languages and provide access to a single truth.

Anahtar Kelimeler

• ChatGPT-3.5
• Copilot
• Gemini
• English and Turkish
• Intraocular tumors and ophthalmic pathologies

Oftalmik Patolojiler ve Göz İçi Tümörlerinde Dil Farklılıklarının Yapay Zeka Chatbot Performansı Üzerindeki Etkisinin Değerlendirilmesi: ChatGPT-3.5, Copilot ve Gemini Üzerine Bir Çalışma

Öz

Amaç: ChatGPT-3,5, Copilot ve Gemini yapay zeka sohbet botlarının oftalmik patolojiler ve intraoküler tümörlerle ilişkili çoktan seçmeli sorularda ki başarısına dil farklılığının etkisini araştırmak Materyal ve Method: Oftalmik patolojiler ve intraoküler tümörlerle ilgili bilgi düzeyini test eden 36 İngilizce soru çalışmaya dahil edildi. Sertifikasyonlu çevirmen (native speaker) tarafından Türkçe çevirilerinin gerçekleştirilmesi sonrasında bu soruların hem İngilizce hem de Türkçe olarak ChatGPT-3,5, Copilot ve Gemini sohbet botlarına soruldu. Verilen cevaplar cevap anahtarı ile karşılaştırılıp doğru ve yanlış olarak gruplandırıldı. Bulgular: ChatGPT-3,5, Copilot ve Gemini İngilizce sorulara sırası ile %75, %66,7 ve %63,9 oranında doğru cevap verdi. Bu programlar Türkçe sorulara ise sırası ile %63,9, %66,7 ve %69,4 oranında doğru cevap verdi. Sohbet botları arasında soruların Türkçe hallerini cevaplamada farklı oranda doğru cevap görüldüğü halde, istatistiksel olarak anlamlı bir fark tespit edilmedi (p>0,05). Sonuç: Yapay zeka sohbet botlarının bilgi dağarcığının geliştirilmesinin yanında farklı dillerde aynı algıyı oluşturabilmek ve tek doğruya erişimi sağlayabilmek için farklı dilleri anlama, çevirebilme ve fikir üretebilme özelliklerinin de geliştirilmeye ihtiyacı vardır.

Anahtar Kelimeler

• ChatGPT-3.5
• Copilot
• Gemini
• Göz içi tümörler ve oftalmik patolojiler
• İngilizce ve Türkçe

Kaynakça

1. 1. Rahimy E. Deep learning applications in ophthalmology. Curr Opin Ophthalmol. 2018;29(3):254-260.
2. 2. Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmo-logy. Br J Ophthalmol. 2019;103(2):167-175.
3. 3. Antaki F, Coussa RG, Kahwati G, Hammamji K, Sebag M, Duval R. Accuracy of automated machine learning in classifying reti-nal pathologies from ultra-widefield pseudocolour fundus images. Br J Ophthalmol. 2023;107(1):90-95.
4. 4. Schmidt-Erfurth U, Sadeghipour A, Gerendas BS, Waldstein SM, Bogunović H. Artificial intelligence in retina. Prog Retin Eye Res. 2018;67:1-29.
5. 5. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language Models are Unsupervised Multitask Learners. Acces-sed June 26, 2023. https://github.com/codelucas/newspaper
6. 6. Syed NA, Berry JL, Heegaard S, Kramer TR, Lee V, Raparia K, et al., eds. Ophthalmic Pathology and Intraocular Tumors. Ame-rican Academy of Ophthalmology; 2023.
7. 7. Wen J, Wang W. The future of ChatGPT in academic research and publishing: A commentary for clinical and translational medicine. Clin Transl Med. 2023;13(3): e1207
8. 8. Tao BKL, Hua N, Milkovich J, Micieli JA. ChatGPT-3.5 and Bing Chat in ophthalmology: an updated evaluation of performan-ce, readability, and informative sources. Eye 2024. Published online March 20, 2024:1-6.

9. 9. Bing Chat | Microsoft Edge. Accessed July 4, 2024. https://www.microsoft.com/en-us/edge/features/bing-chat?form=MT00D8
10. 10. Waisberg E, Ong J, Masalkhi M, Zaman N, Sarker P, Lee AG, et al. Google’s AI chatbot “Bard”: a side-by-side comparison with ChatGPT and its utilization in ophthalmology. Eye 2023 38:4. 2023;38(4):642-645.
11. 11. Google AI updates: Bard and new AI features in Search. Acces-sed July 4, 2024. https://blog.google/technology/ai/bard-google-ai-search-updates/
12. 12. Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci. 2023;39(2):605.
13. 13. Jeblick K, Schachtner B, Dexl J, Mittermeier A, Stüber AT, Topalis J, et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol. 2024;34(5):2817-2825.
14. 14. Korngiebel DM, Mooney SD. Considering the possibilities and pitfalls of Generative Pre-trained Transformer 3 (GPT-3) in he-althcare delivery. npj Digital Medicine 2021 4:1. 2021;4(1):1-3.
15. 15. Nath S, Marie A, Ellershaw S, Korot E, Keane PA. New meaning for NLP: the trials and tribulations of natural language proces-sing with GPT-3 in ophthalmology. British Journal of Ophthal-mology. 2022;106(7):889-892.
16. 16. Haddad F, Saade JS. Performance of ChatGPT on Ophthalmo-logy-Related Questions Across Various Examination Levels: Ob-servational Study. JMIR Med Educ. 2024;10:e50842.
17. 17. Moshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial Intelligence in Ophthalmology: A Comparative Analy-sis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions. Cureus. 2023;15(6):e40822.
18. 18. Canleblebici M, Dal A, Erdağ M. Evaluation of the Performance of Large Language Models (ChatGPT-3.5, ChatGPT-4, Bing and Bard) in Turkish Ophthalmology Chief-Assistant Exams: A Com-parative Study. Turkiye Klinikleri J Ophthalmol. Published on-line June 11, 2024.
19. 19. Mihalache A, Grad J, Patil NS, Huang RS, Popovic MM, Malli-patna A, et al. Google Gemini and Bard artificial intelligence chatbot performance in ophthalmology knowledge assess-ment. Eye 2024. Published online April 13, 2024:1-6.