Readability of Patient Education Texts Generated by Artificial Intelligence Chatbots Across Dental and Medical Topics

Öz

Objective: The increasing use of artificial intelligence (AI)–based chatbots in health information seeking has raised concerns regarding the readability of the generated content. This study aimed to compare the readability of AI-generated patient information texts across different chatbot models, content domains (dental and medical), and content types (diagnosis and treatment). Methods: In this cross-sectional descriptive study, patient-oriented texts generated by four AI systems (ChatGPT, Gemini, Copilot, and DeepSeek) were analyzed. Eight topics representing dental and medical fields were selected using Google Trends. For each topic, standardized prompts were used to generate diagnosis- and treatment-related content. Readability was assessed using the Flesch Reading Ease Score (FRES) and the Flesch–Kincaid Grade Level (FKGL). Statistical analyses were performed using Friedman and Mann–Whitney U tests, with the significance level set at P < .05. Results: No statistically significant differences were found among chatbot models in terms of FRES and FKGL values (P = .702 and P = .226, respectively). Similarly, no significant differences were observed between dental and medical content (P = .460 and P = .732, respectively). However, diagnosis-related texts demonstrated significantly higher FRES values compared to treatment-related texts (P = .016), indicating better readability. No significant difference was found between the two content types in terms of FKGL (P = .062). Conclusion: AI-generated patient information texts exhibit similar readability levels across chatbot models and generally exceed recommended reading levels for patient education. While content domain does not significantly affect readability, content type may influence certain readability metrics. These findings suggest that current AI systems do not consistently produce patient-friendly text and highlight the need for improving the clarity and accessibility of AI-generated health information.

Anahtar Kelimeler

• Artificial Intelligence
• Comprehension
• Natural Language Processing
• Patient Education as Topic
• Health Literacy.

Diş Hekimliği ve Tıbbi Konularda Yapay Zekâ Chatbotları Tarafından Oluşturulan Hasta Eğitimi Metinlerinin Okunabilirliği

Öz

Amaç: Yapay zekâ tabanlı chatbotların sağlıkla ilgili bilgi arama sürecinde artan kullanımı, üretilen içeriklerin okunabilirliği konusunda önemli sorular ortaya çıkarmaktadır. Bu çalışmanın amacı, yapay zekâ tarafından üretilen hasta bilgilendirme metinlerinin okunabilirliğini farklı chatbot modelleri, içerik alanı (dental ve medikal) ve içerik türü (tanı ve tedavi) açısından karşılaştırmaktır. Yöntem: Bu kesitsel tanımlayıcı çalışmada, dört farklı yapay zekâ sistemi (ChatGPT, Gemini, Copilot ve DeepSeek) tarafından üretilen hasta odaklı metinler analiz edilmiştir. Dental ve medikal alanları temsil eden sekiz konu Google Trends kullanılarak belirlenmiştir. Her konu için tanı ve tedaviye yönelik standartlaştırılmış istemler kullanılmıştır. Okunabilirlik, Flesch Reading Ease Score (FRES) ve Flesch–Kincaid Grade Level (FKGL) ile değerlendirilmiş; istatistiksel analizlerde Friedman ve Mann–Whitney U testleri kullanılmıştır. Anlamlılık düzeyi P < .05 olarak kabul edilmiştir. Bulgular: Chatbot modelleri arasında FRES ve FKGL değerleri açısından istatistiksel olarak anlamlı bir fark bulunmamıştır (sırasıyla P = .702 ve P = .226). Benzer şekilde, dental ve medikal içerikler arasında da anlamlı fark saptanmamıştır (sırasıyla P = .460 ve P = .732). Buna karşılık, tanı içeriklerinin FRES değerleri tedavi içeriklerine göre anlamlı derecede daha yüksek bulunmuştur (P = .016). FKGL açısından içerik türleri arasında anlamlı fark gözlenmemiştir (P = .062). Sonuç: Yapay zekâ tarafından üretilen hasta bilgilendirme metinlerinin okunabilirliği chatbot modellerinden bağımsız olarak benzer düzeydedir ve genel olarak önerilen hasta okuma seviyelerinin üzerindedir. İçerik alanı okunabilirliği anlamlı düzeyde etkilemezken, içerik türü belirli ölçütlerde farklılık oluşturabilmektedir. Bu bulgular, mevcut yapay zekâ sistemlerinin hasta dostu metin üretiminde sınırlılıklar taşıdığını ve sağlık içeriklerinin daha anlaşılır hale getirilmesine yönelik geliştirmelere ihtiyaç olduğunu göstermektedir.

Anahtar Kelimeler

• Yapay Zekâ
• Anlama
• Doğal Dil İşleme
• Hasta Eğitimi
• Sağlık Okuryazarlığı

Kaynakça

1. Acar, N. K., Sengul, F., Bardakci, E., & Celikel, P. (2026). Assessing the accuracy, reliability, quality, and readability of artificial intelligence chatbots in patient education: insights from zirconia crowns. BMC Oral Health, 26, 502. https://doi.org/10.1186/s12903-026-07884-9
2. Behers, B. J., Vargas, I. A., Behers, B. M., Rosario, M. A., Wojtas, C. N., Deevers, A. C., Hamad, K. M. (2024). Assessing the readability of patient education materials on cardiac catheterization from artificial intelligence chatbots: an observational cross-sectional study. Cureus, 16(7), e63865. https://doi.org/10.7759/cureus.63865
3. Carlson, J. A., Cheng, R. Z., Lange, A., Nagalakshmi, N., Rabets, J., Shah, T., & Sindhwani, P. (2024). Accuracy and readability of artificial intelligence chatbot responses to vasectomy-related questions: public beware. Cureus, 16(8), e67996. https://doi.org/10.7759/cureus.67996
4. DeTemple, D. E., & Meine, T. C. (2025). Comparison of the readability of ChatGPT and Bard in medical communication: a meta-analysis. BMC Medical Informatics and Decision Making, 25(1), 325. https://doi.org/10.1186/s12911-025-03035-2
5. Eid, K., Eid, A., Wang, D., Raiker, R. S., Chen, S., & Nguyen, J. (2024). Optimizing ophthalmology patient education via chatbot-generated materials: readability analysis of AI-generated patient education materials and the American Society of Ophthalmic Plastic and Reconstructive Surgery patient brochures. Ophthalmic Plastic & Reconstructive Surgery, 40(2), 212–216. https://doi.org/10.1097/IOP.0000000000002549
6. Eltorai, A. E., Ghanian, S., Adams, C. A., Jr., Born, C. T., & Daniels, A. H. (2014). Readability of patient education materials on the American Association for Surgery of Trauma website. Archives of Trauma Research, 3(2), e18161. https://doi.org/10.5812/atr.18161
7. Halboub, E., Hakami, R. A. M., Khalufi, K. N. A., Hakami, S. A. H., & Alhajj, M. N. (2025). Assessment of quality, understandability, actionability, and readability of responses of selected chatbots to the top searched queries about oral cancer. Digital Dentistry Journal, 1(1), 100008. https://doi.org/10.1016/j.ddj.2025.100008
8. Hamzeh, N., Lidder, A. K., Feder, R. S., Sarmiento, E. A., Mirza, R. G., Thau, A. J., & Tanna, A. P. (2026). Accuracy and readability of Chat Generative Pre-Trained Transformer-4 Omni in answering ophthalmology patient questions. Ophthalmology Science, 6(2), 101007. https://doi.org/10.1016/j.xops.2025.101007

9. Helvacioglu-Yigit, D., Demirturk, H., Ali, K., Tamimi, D., Koenig, L. J., & Almashraqi, A. (2025). Evaluating artificial intelligence chatbots for patient education in oral and maxillofacial radiology. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, 139(6), 750–759. https://doi.org/10.1016/j.oooo.2025.01.001
10. Høj, S., & Meteran, H. (2025). Correspondence regarding “Readability of rhinology-related patient education materials generated by artificial intelligence-based chatbots”. International Forum of Allergy & Rhinology, 15, 568–569. https://doi.org/10.1002/alr.23585
11. Kincaid, J. P., Fishburne, R. P., Jr., Rogers, R. L., & Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command Research Branch, 51. https://doi.org/10.21236/ADA006655
12. Liu, Z., Zou, W., & Lin, C. (2025). Exploring the influence of privacy concerns, AI literacy, and perceived health stigma on AI chatbot use in healthcare: an uncertainty reduction approach. Patient Education and Counseling, 140, 109271. https://doi.org/10.1016/j.pec.2025.109271
13. Maywald, L., Nguyen, L., Winterstein, J. T., Hetz, M. J., Mangold, M. H., Renner, L. V., Brinker, T. J., Wessels, F., & Carl, N. (2025). Readability of chatbot responses in prostate cancer and urological care: objective metrics versus patient perceptions. Current Oncology, 32(10), 582. https://doi.org/10.3390/curroncol32100582
14. Miftaroski, A., Zowalla, R., Wiesner, M., & Pobiruchin, M. (2026). Leveraging large language models to improve the readability of German online medical texts: evaluation study. JMIR AI, 5, e77149. https://doi.org/10.2196/77149
15. Musheyev, D., Pan, A., Gross, P., Kamyab, D., Kaplinsky, P., Spivak, M., Bragg, M. A., Loeb, S., & Kabarriti, A. E. (2024). Readability and information quality in cancer information from a free vs paid chatbot. The Journal of the American Medical Association Network Open, 7(7), e2422275. https://doi.org/10.1001/jamanetworkopen.2024.22275
16. Okuhara, T., Furukawa, E., Okada, H., Yokota, R., & Kiuchi, T. (2025). Readability of written information for patients across 30 years: a systematic review of systematic reviews. Patient Education and Counseling, 135, 108656. https://doi.org/10.1016/j.pec.2025.108656
17. Olszewski, R., Brzeziński, J., Watros, K., & Rysz, J. (2026). Quantifying readability in chatbot-generated medical texts using classical linguistic indices: a review. Applied Sciences, 16(3), 1423. https://doi.org/10.3390/app16031423
18. Ozduran, E., Akkoc, I., Büyükçoban, S., Erkin, Y., & Hanci, V. (2025). Readability, reliability and quality of responses generated by ChatGPT, Gemini, and Perplexity for the most frequently asked questions about pain. Medicine, 104(11), e41780. https://doi.org/10.1097/MD.0000000000041780
19. Stephenson-Moe, C. A., Behers, B. J., Gibons, R. M., Behers, B. M., Herrera, L. D. J., Anneaud, D., Rosario, M. A., Wojtas, C. N., Bhambrah, S., & Hamad, K. M. (2025). Assessing the quality and readability of patient education materials on chemotherapy cardiotoxicity from artificial intelligence chatbots: an observational cross-sectional study. Medicine, 104(15), e42135. https://doi.org/10.1097/MD.0000000000042135
20. Zaleski, A. L., Berkowsky, R., Craig, K. J. T., & Pescatello, L. S. (2024). Comprehensiveness, accuracy, and readability of exercise recommendations provided by an AI-based chatbot: mixed methods study. JMIR Medical Education, 10(1), e51308. https://doi.org/10.2196/51308