Kuru Göz Hastalığı Hakkında Sık Sorulan Sorulara Büyük Dil Modeli Sohbet Botlarının Yanıtlarının Uygunluğu ve Okunabilirliği: Kesitsel Araştırma

Öz

ÖZ Amaç: Büyük dil modeli sohbet botları tıbbi bilgi için giderek daha fazla başvurulan kaynaklar haline gelmiştir. Bu çalışma, kuru göz hastalığına ilişkin sık sorulan hasta sorularına verilen yanıtların doğruluğunu ve okunabilirliğini değerlendirmeyi amaçladı. Yöntemler: Bu kesitsel çalışmada dört sohbet botunun (ChatGPT 3.5, ChatGPT 4.0, Google Gemini ve Microsoft Copilot) kuru göz hastalığı ile ilgili elli standart soruya verdikleri yanıtlar analiz edildi. İki oftalmolog yanıtların doğruluğunu beş basamaklı Likert ölçeğiyle bağımsız olarak puanladı ve değerlendiriciler arası uyum Cohen’in kappa katsayısı ile ölçüldü. Okunabilirlik; Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman–Liau Index, Simple Measure of Gobbledygook, Flesch Reading Ease ve Reach skoru ile değerlendirildi. İstatistiksel analizlerde tekrarlayan ölçümlerde varyans analizi veya Friedman testi ve Bonferroni düzeltmesi kullanıldı. Bulgular: Değerlendiriciler arası uyum mükemmeldi (kappa = 0,88). En yüksek doğruluk Google Gemini’de (4,84 ± 0,37) izlendi; bunu Microsoft Copilot (4,76 ± 0,48), ChatGPT 4.0 (4,42 ± 0,50) ve ChatGPT 3.5 (4,32 ± 0,59; p < 0,001) takip etti. Gemini ve Copilot her iki ChatGPT versiyonuna anlamlı olarak üstün bulundu. Okunabilirlik anlamlı farklılık gösterdi (p < 0,001). ChatGPT 4.0 en basit metinleri üreterek en düşük sınıf düzeyleri, en yüksek Flesch Okuma Kolaylığı ve en geniş Reach skoruna ulaştı. Gemini ve ChatGPT 3.5 daha karmaşık yanıtlar üretirken Copilot orta düzey değerler sergiledi. Sonuçlar: Sohbet botları tamamlayıcı güçlü yönler ortaya koydu. Gemini ve Copilot en yüksek doğruluğa sahipken, ChatGPT 4.0 en okunabilir ve erişilebilir yanıtları sundu. Sohbet botları kuru göz hastalığında hasta eğitimi için faydalı olabilir; ancak profesyonel gözetim gerekliliğini korumaktadır.

Anahtar Kelimeler

• Yapay zeka
• Sohbet botları
• Kuru göz hastalığı
• Büyük dil modelleri
• Okunabilirlik

Appropriateness and Readability of Large Language Model Chatbot Responses to Frequently Asked Questions About Dry Eye Disease: Cross-Sectional Study

Öz

ABSTRACT Objective: Large language model chatbots are increasingly consulted for medical information. This study evaluated the accuracy and readability of chatbot responses to common patient questions on dry eye disease. Methods: This cross-sectional study analysed responses from four chatbots (ChatGPT 3.5, ChatGPT 4.0, Google Gemini, and Microsoft Copilot) to fifty standardised questions about dry eye disease. Two ophthalmologists independently rated accuracy on a five-point Likert scale, with inter-rater agreement measured by Cohen’s kappa. Readability was assessed using Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman–Liau Index, Simple Measure of Gobbledygook, Flesch Reading Ease, and Reach score. Statistical tests included repeated-measures analysis of variance or Friedman tests with Bonferroni correction. Results: Agreement between raters was excellent (kappa = 0.88). Google Gemini showed the highest accuracy (4.84 ± 0.37), followed by Microsoft Copilot (4.76 ± 0.48), ChatGPT 4.0 (4.42 ± 0.50), and ChatGPT 3.5 (4.32 ± 0.59; p < 0.001). Gemini and Copilot significantly outperformed both ChatGPT versions. Readability differed significantly (p < 0.001). ChatGPT 4.0 produced the simplest texts, with the lowest grade levels, highest Flesch Reading Ease and broadest Reach. Gemini and ChatGPT 3.5 generated more complex responses, while Copilot showed intermediate values. Conclusions: Chatbots demonstrated complementary strengths. Gemini and Copilot were most accurate, whereas ChatGPT 4.0 was most readable and accessible. Chatbots may aid patient education in dry eye disease, but professional oversight remains necessary.

Anahtar Kelimeler

• Artificial intelligence
• Chatbots
• Dry eye disease
• Large language models
• Readability

Kaynakça

1. Pflugfelder SC, de Paiva CS. The Pathophysiology of Dry Eye Disease: What We Know and Future Directions for Research. Ophthalmol 2017;124(11S):S4-S13.
2. Pflugfelder SC, De Paiva CS, Villarreal AL, Stern ME. Effects of sequential artificial tear and cyclosporine emulsion therapy on conjunctival goblet cell density and transforming growth factor- beta2 production. Cornea 2008; 27(1):64–9.
3. Stapleton F, Alves M, Bunya VY, Jalbert I, Lekhanont K, Malet F, Na KS, Schaumberg D, Uchino M, Vehof J, Viso E, Vitale S, Jones L. TFOS DEWS II epidemiology report. Ocul surf 2017;15(3):334-365.
4. Swoboda CM, Van Hulle JM, McAlearney AS, Huerta TR. Odds of talking to healthcare providers as the initial source of healthcare information: updated cross-sectional results from the Health Information National Trends Survey (HINTS). BMC Fam Pract. 2018;19(1):146.
5. Rossettini G, Rodeghiero L, Corradi F, Cook C, Pillastrini P, Turolla A, Castellini G, Chiappinotto S, Gianola S, Palese A. Comparative accuracy of ChatGPT-4, Microsoft Copilot and Google Gemini in the Italian entrance test for healthcare sciences degrees: a cross-sectional study. BMC Med Educ. 2024;24(1):694.
6. Aydın FO, Aksoy BK, Ceylan A, Akbaş YB, Ermiş S, Kepez Yıldız B, Yıldırım Y. Readability and Appropriateness of Responses Generated by ChatGPT 3.5, ChatGPT 4.0, Gemini, and Microsoft Copilot for FAQs in Refractive Surgery. Turk J Ophthalmol. 2024;54(6):313-317.
7. Nichani PAH, Ong Tone S, AlShaker SM, Teichman JC, Chan CC. Use of Online Large Language Model Chatbots in Cornea Clinics. Cornea 2024;44(6):788-794.
8. Owens OL, Leonard M. A Comparison of Prostate Cancer Screening Information Quality on Standard and Advanced Versions of ChatGPT, Google Gemini, and Microsoft Copilot: A Cross-Sectional Study. Am J Health Promot. 2025;39(5):766-776.

9. Sevgi, M., Antaki, F., & Keane, P. A. Medical education with large language models in ophthalmology: custom instructions and enhanced retrieval capabilities. Br J Ophthalmol. 2024;108(10):1354–1361.
10. Raddatz N, Kettinger W, Coyne J. Giving to Get Well: Patients’ Willingness to Manage and Share Health Information on AI-Driven Platforms. Communications of the Association for Information Systems. 2023;52:1017-1049.
11. Dihan QA, Brown AD, Chauhan MZ, Alzein AF, Abdelnaem SE, Kelso SD, Rahal DA, Park R, Ashraf M, Azzam A, Morsi M, Warner DB, Sallam AB, Saeed HN, Elhusseiny AM. Leveraging large language models to improve patient education on dry eye disease. Eye (Lond). 2025;39(6):1115-1122.
12. Chen X, Zhao Z, Zhang W, Xu P, Wu Y, Xu M, Gao L, Li Y, Shang X, Shi D, He M. EyeGPT for Patient Inquiries and Medical Education: Development and Validation of an Ophthalmology Large Language Model. J Med Internet Res. 2024;26:e60063.
13. Şensoy E, Çıtırık M. Performance of ChatGPT-3.5, Copilot and Gemini in answering English and Turkish questions related to ocular surface diseases and cornea: A comparison study. Turkish Journal of Clinical and Experimental Ophthalmology. 2025;20(6).
14. Wang J, Shi R, Le Q, Shan K, Chen Z, Zhou X, He Y, Hong J. Evaluating the effectiveness of large language models in patient education for conjunctivitis. Br J Ophthalmol. 2025;109(2):185-191.
15. Masoni M, Guelfi MR. Going beyond the concept of readability to improve comprehension of patient education materials. Intern Emerg Med. 2017;12(4):531-533.
16. Shi R, Liu S, Xu X, Ye Z, Yang J, Le Q, Qiu J, Tian L, Wei A, Shan K, Zhao C, Sun X, Zhou X, Hong J. Benchmarking four large language models' performance of addressing Chinese patients' inquiries about dry eye disease: A two-phase study. Heliyon. 2024;10(14):e34391.
17. Y ang Z, Wang D, Zhou F, Song D, Zhang Y, Jiang J, Kong K, Liu X, Qiao Y, Chang RT, Han Y, Li F, Tham CC, Zhang X. Understanding natural language: Potential application of large language models to ophthalmology. Asia Pac J Ophthalmol (Phila). 2024;13(4):100085.
18. Huang AS, Hirabayashi K, Barna L, Parikh D, Pasquale LR. Assessment of a Large Language Model's Responses to Questions and Cases About Glaucoma and Retina Management. JAMA Ophthalmol. 2024;142(4):371-5.