Is ChatGPT a Useful Tool for Ophthalmology Practice?

Yıl 2024, Cilt: 8 Sayı: 3, 221 - 227, 30.12.2024

Fuat Yavrum , Dilara Özkoyuncu

https://doi.org/10.30565/medalanya.1531790

Öz

Aim: This study aimed to assess ChatGPT-3.5's performance in ophthalmology, comparing its responses to clinical case-based and multiple-choice (MCQ) questions.

Methods: ChatGPT-3.5, an AI model developed by OpenAI, was employed. It responded to 98 case-based questions from "Ophthalmology Review: A Case-Study Approach" and 643 MCQs from "Review Questions in Ophthalmology" book. ChatGPT's answers were compared to the books, and statistical analysis was conducted.

Results: ChatGPT achieved an overall accuracy of 56.1% in case-based questions. Accuracy varied across categories, with the highest in the retina section (69.5%) and the lowest in the trauma section (38.2%). In MCQ, ChatGPT's accuracy was 53.5%, with the weakest in the optics section (32.6%) and the highest in pathology and uveitis (66.7% and 63.0%, respectively). ChatGPT performed better in case-based questions in the retina and pediatric ophthalmology sections than MCQ.

Conclusion: ChatGPT-3.5 exhibits potential as a tool in ophthalmology, particularly in retina and pediatric ophthalmology. Further research is needed to evaluate ChatGPT's clarity and acceptability for open-ended questions.

Anahtar Kelimeler

Artificial intelligence, ChatGPT, Large language model, Ophthalmology

ChatGPT Oftalmoloji Pratiğinde Faydalı Bir Araç Mıdır?

Öz

Amaç: ChatGPT-3.5'in performansını göz hastalıkları alanında değerlendirmek, klinik vaka bazlı sorular ve çoktan seçmeli sorulara (ÇSS) verdiği yanıtların doğruluk oranını karşılaştırmaktır.

Yöntem: Çalışmada OpenAI tarafından geliştirilen bir yapay zeka modeli olan ChatGPT-3.5 kullanıldı. Modelden, "Ophthalmology Review: A Case-Study Approach" kitabından 98 vaka bazlı soruya ve "Review Questions in Ophthalmology" kitabından 643 ÇSS'ye yanıt vermesi istendi. ChatGPT'nin cevapları kitaplarla karşılaştırıldı ve istatistiksel analizi yapıldı.

Bulgular: ChatGPT, vaka bazlı sorularda genel olarak %56,1 doğruluk oranı gösterdi.. Doğruluk oranı kategoriler arasında en yüksek retina bölümünde (%69,5) ve en düşük travma bölümünde (%38,2) idi. ÇSS'de ChatGPT'nin genel doğruluk oranı %53,5 olarak gözlendi, bunların en düşüğü optik bölümünde (%32,6) ve en yükseği patoloji ve üveit bölümlerinde (%66,7 ve %63) idi. ChatGPT özellikle retina ve pediatrik oftalmoloji bölümlerindeki vaka bazlı sorularda ÇSS’ye kıyasla daha iyi performans gösterdi.

Sonuç: ChatGPT-3.5, özellikle retina ve pediatrik oftalmoloji alanlarında göz hastalıkları için potansiyel bir yardımcı araç olarak görülmektedir. ChatGPT'nin açık uçlu sorular için netlik ve kabul edilebilirliğini değerlendirmek için daha fazla araştırma yapılması gerekmektedir.

Anahtar Kelimeler

Yapay zeka, ChatGPT, Büyük dil modeli, Oftalmoloji

Kaynakça

• 1. Li JO, Liu H, Ting DSJ, Jeon S, Chan RVP, et al. Digital technology, tele-medicine and artificial intelligence in ophthalmology: A global perspective. Prog Retin Eye Res. 2021;82:100900. doi: 10.1016/j.preteyeres.2020.100900.
• 2. Sallam M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare (Basel). 2023;11(6):887. doi: 10.3390/healthcare11060887.
• 3. Introducing ChatGPT. https://openai.com/blog/chatgpt Accessed May 17, 2023.
• 4. Alkaissi H, McFarlane SI. Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus. 2023;15(2):e35179. doi: 10.7759/cureus.35179.
• 5. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023;9:e45312. doi: 10.2196/57594.
• 6. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2(2):e0000198. doi: 10.1371/journal.pdig.0000198.
• 7. Cai LZ, Shaheen A, Jin A, Fukui R, Yi JS, et al. Performance of Generative Large Language Models on Ophthalmology Board-Style Questions. Am J Ophthalmol. 2023;254:141-9. doi: 10.1016/j.ajo.2023.05.024.
• 8. Raumviboonsuk P, Krause J, Chotcomwongse P, Sayres R, Raman R, et al. Deep learning versus human graders for classifying diabetic retinopathy severity in a nationwide screening program. NPJ Digit Med. 2019;2:25. doi: 10.1038/s41746-019-0099-8.
• 9. Bogunovic H, Montuoro A, Baratsits M, Karantonis MG, Waldstein SM, et al. Machine Learning of the Progression of Intermediate Age-Related Macular Degeneration Based on OCT Imaging. Invest Ophthalmol Vis Sci. 2017;58(6):BIO141-50. doi: 10.1167/iovs.17-21789.
• 10. Singh K, Smiddy WE, Lee AG. Ophthalmology review : a case-study approach. Second edition. Thieme; 2018
• 11. Kenneth C. Chern, Michael A. Saidel. Review Questions in Ophthalmology. Third edition. Wolters Kluwer; 2014
• 12. Antaki F, Touma S, Milad D, El-Khoury J, Duval R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmol Sci. 2023;3(4):100324. doi: 10.1016/j.xops.2023.100324.
• 13. Mihalache A, Popovic MM, Muni RH. Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmol. 2023;141(6):589-97. doi: 10.1001/jamaophthalmol.2023.1144.
• 14. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi: 10.1038/s41586-023-06291-2.
• 15. Moshirfar M, Altaf AW, Stoakes IM, Tuttle JJ, Hoopes PC. Artificial Intelligence in Ophthalmology: A Comparative Analysis of GPT-3.5, GPT-4, and Human Expertise in Answering StatPearls Questions. Cureus. 2023;15(6):e40822. doi: 10.7759/cureus.40822.
• 16. Stunkel L, Mackay DD, Bruce BB, Newman NJ, Biousse V. Referral Patterns in Neuro-Ophthalmology. J Neuroophthalmol. 2020;40(4):485-93. doi: 10.1097/WNO.0000000000000846.