Artificial Intelligence Chatbots as Patient Information Tools: A Bilingual, Two-Round Evaluation of the e-Nabız Personal Health Record System

Öz

This study evaluated four widely used large language models—ChatGPT, Claude, Gemini, and DeepSeek—as patient-information tools for e-Nabız, the national personal health record system of Türkiye, and tested whether their answers were reproducible one week apart. Thirty authentic patient questions spanning seven e-Nabız domains were submitted to each model in Turkish and English, in fresh sessions with default settings, and the protocol was repeated after one week (480 responses). A dated copy of the official e-Nabız User Guide served as the gold standard, and responses were scored blind for accuracy, completeness, the Global Quality Score, coverage, and safety, with readability computed using the Flesch, Flesch–Kincaid, Ateşman, and Bezirci–Yılmaz formulas. Model performance differed significantly in both languages, with Claude and Gemini outperforming ChatGPT and DeepSeek; two models performed better in Turkish. All models wrote above recommended reading levels, and although aggregate quality was stable between rounds, individual answers were substantially reworded and sometimes materially changed. Chatbots should complement, not replace, official e-Nabız guidance.

Anahtar Kelimeler

• e-Nabız
• large language models
• patient information
• digital health literacy
• readability

Hasta Bilgilendirme Aracı Olarak Yapay Zekâ Sohbet Botları: e-Nabız Kişisel Sağlık Kaydı Sisteminin İki Dilli ve İki Turlu Değerlendirmesi

Öz

Bu çalışma, yaygın olarak kullanılan dört büyük dil modelinin—ChatGPT, Claude, Gemini ve DeepSeek—Türkiye’nin ulusal kişisel sağlık kaydı sistemi e-Nabız’a ilişkin hasta sorularını hasta bilgilendirme aracı olarak yanıtlamadaki performansını ve bu yanıtların bir hafta arayla tekrarlanabilirliğini değerlendirmiştir. Yedi e-Nabız alanını kapsayan 30 gerçek hasta sorusu, her modele Türkçe ve İngilizce olarak, varsayılan ayarlarla ve yeni oturumlarda yöneltilmiş; protokol bir hafta sonra tekrarlanmıştır (toplam 480 yanıt). Altın standart olarak resmi e-Nabız Kullanım Kılavuzu kullanılmış; yanıtlar model kimliğine kör biçimde doğruluk, eksiksizlik, Global Kalite Skoru, kapsam ve güvenlik açısından puanlanmış, okunabilirlik Flesch, Flesch–Kincaid, Ateşman ve Bezirci–Yılmaz formülleriyle hesaplanmıştır. Model performansı her iki dilde de anlamlı biçimde farklılaşmış; Claude ve Gemini, ChatGPT ve DeepSeek’i geride bırakmış, iki model Türkçede daha iyi performans göstermiştir. Tüm modeller önerilen okunabilirlik düzeyinin üzerinde metin üretmiş; turlar arasında toplam kalite değişmezken bireysel yanıtlar büyük ölçüde yeniden ifade edilmiş ve zaman zaman içerik olarak değişmiştir. Sohbet botları resmi e-Nabız rehberliğinin yerine değil, tamamlayıcısı olarak konumlandırılmalıdır.

Anahtar Kelimeler

• e-Nabız
• büyük dil modelleri
• hasta bilgilendirme
• dijital sağlık okuryazarlığı
• okunabilirlik

Kaynakça

1. Altuntaş, M. C., & Erdinç Akyol, G. (2026). A bilingual evaluation of chatbot performance in bruxism-related information: Accuracy and readability across two models. BMC Oral Health, 26(1), Article 30. https://doi.org/10.1186/s12903-026-08581-3
2. Aydin, G., Kumru, S., & Balin, H. (2026). Digital health literacy as a driver of continued electronic health record use: Insights from necessity and sufficiency analyses. Informatics for Health and Social Care, 1–21. https://doi.org/10.1080/17538157.2026.2676934
3. Ateşman, E. (1997). Türkçede okunabilirliğin ölçülmesi. Dil Dergisi, 58, 71–74.
4. Balel, Y. (2023). Can ChatGPT be used in oral and maxillofacial surgery? Journal of Stomatology, Oral and Maxillofacial Surgery, 124(5), Article 101471. https://doi.org/10.1016/j.jormas.2023.101471
5. Bernard, A., Langille, M., Hughes, S., Rose, C., Leddin, D., & Veldhuyzen van Zanten, S. (2007). A systematic review of patient inflammatory bowel disease information resources on the World Wide Web. American Journal of Gastroenterology, 102(9), 2070–2077. https://doi.org/10.1111/j.1572-0241.2007.01325.x
6. Bezirci, B., & Yılmaz, A. E. (2010). Metinlerin okunabilirliğinin ölçülmesi üzerine bir yazılım kütüphanesi ve Türkçe için yeni bir okunabilirlik ölçütü. Dokuz Eylül Üniversitesi Mühendislik Fakültesi Fen ve Mühendislik Dergisi, 12(3), 49–62.
7. Birinci, Ş. (2023). A digital opportunity for patients to manage their health: Turkey national personal health record system (the e-Nabız). Balkan Medical Journal, 40(3), 215–221. https://doi.org/10.4274/balkanmedj.galenos.2023.2023-2-77
8. Charnock, D., Shepperd, S., Needham, G., & Gann, R. (1999). DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. Journal of Epidemiology and Community Health, 53(2), 105–111. https://doi.org/10.1136/jech.53.2.105

9. Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., … Logullo, P. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, Article e078378. https://doi.org/10.1136/bmj-2023-078378
10. Çörekci, A. M., Ateş, B., Öztürk, A. A., & Özdemir, M. (2026). Performance of three large language models in answering parent-focused questions on rickets. BMC Pediatrics, 26(1), Article 6. https://doi.org/10.1186/s12887-026-06851-1
11. Esmailpour, H., Rasaie, V., Babaee Hemmati, Y., & Falahchai, M. (2025). Performance of artificial intelligence chatbots in responding to patient queries related to dental prostheses. BMC Oral Health, 25(1), Article 574. https://doi.org/10.1186/s12903-025-05965-9
12. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017), 625–630. https://doi.org/10.1038/s41586-024-07421-0
13. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233. https://doi.org/10.1037/h0057532
14. Gallifant, J., Afshar, M., Ameen, S., Aphinyanaphongs, Y., Chen, S., Cacciamani, G., … Bitterman, D. S. (2025). The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine, 31(1), 60–69. https://doi.org/10.1038/s41591-024-03425-5
15. Gibson, D., Jackson, S., Shanmugasundaram, R., Seth, I., Siu, A., Ahmadi, N., … Patel, M. I. (2024). Evaluating the efficacy of ChatGPT as a patient education tool in prostate cancer: Multimetric assessment. Journal of Medical Internet Research, 26, Article e55939. https://doi.org/10.2196/55939
16. Giannakopoulos, K., Kavadella, A., Aaqel Salim, A., Stamatopoulos, V., & Kaklamanos, E. G. (2023). Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: Comparative mixed methods study. Journal of Medical Internet Research, 25, Article e51580. https://doi.org/10.2196/51580
17. Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., … Guyatt, G. (2025). Reporting guideline for chatbot health advice studies: The Chatbot Assessment Reporting Tool (CHART) statement. Annals of Family Medicine, 23(5), 389–398. https://doi.org/10.1370/afm.250386
18. Iqbal, U., Tanweer, A., Rahmanti, A. R., Greenfield, D., Lee, L. T.-J., & Li, Y.-C. J. (2025). Impact of large language model (ChatGPT) in healthcare: An umbrella review and evidence synthesis. Journal of Biomedical Science, 32(1), Article 45. https://doi.org/10.1186/s12929-025-01131-z
19. Kim, Y., Shin, S., & Yoo, S. H. (2025). Performance of large language models in non-English medical ethics-related multiple choice questions: Comparison of ChatGPT performance across versions and languages. BMC Medical Ethics, 26(1), Article 168. https://doi.org/10.1186/s12910-025-01316-z
20. Kufta, A. Y., & Djalilian, A. R. (2025). Enhancing patient education with AI: A readability analysis of AI-generated versus American Academy of Ophthalmology online patient education materials. Journal of Clinical Medicine, 14(19), Article 6968. https://doi.org/10.3390/jcm14196968
21. Künzle, P., & Paris, S. (2024). Performance of large language artificial intelligence models on solving restorative dentistry and endodontics student assessments. Clinical Oral Investigations, 28(11), Article 575. https://doi.org/10.1007/s00784-024-05968-w
22. Laymouna, M., Ma, Y., Lessard, D., Schuster, T., Engler, K., & Lebouché, B. (2024). Roles, users, benefits, and limitations of chatbots in health care: Rapid review. Journal of Medical Internet Research, 26, Article e56930. https://doi.org/10.2196/56930
23. Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., & Denniston, A. K. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI extension. Nature Medicine, 26(9), 1364–1374. https://doi.org/10.1038/s41591-020-1034-x
24. Mirzaoğlu, Ç., Ulutaş, Z., & Karaca, Y. (2026). ChatGPT response consistency to the 2025 ESC/EACTS guidelines for the management of valvular heart disease: A test–retest study using binary and multiple-choice questions. Digital Health, 12. https://doi.org/10.1177/20552076261458145
25. Moulaei, K., Yadegari, A., Baharestani, M., Farzanbakhsh, S., Sabet, B., & Reza Afrash, M. (2024). Generative artificial intelligence in healthcare: A scoping review on benefits, challenges and applications. International Journal of Medical Informatics, 188, Article 105474. https://doi.org/10.1016/j.ijmedinf.2024.105474
26. Moult, B., Franck, L. S., & Brady, H. (2004). Ensuring quality information for patients: Development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expectations, 7(2), 165–175. https://doi.org/10.1111/j.1369-7625.2004.00273.x
27. Nastasi, A. J., Courtright, K. R., Halpern, S. D., & Weissman, G. E. (2023). A vignette-based evaluation of ChatGPT’s ability to provide appropriate and equitable medical advice across care contexts. Scientific Reports, 13(1), Article 17885. https://doi.org/10.1038/s41598-023-45223-y
28. Omar, M., Sorin, V., Collins, J. D., Reich, D., Freeman, R., Charney, A., … Klang, E. (2025). Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Communications Medicine, 5(1), Article 330. https://doi.org/10.1038/s43856-025-01021-3
29. Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025). An empirical study of the non-determinism of ChatGPT in code generation. ACM Transactions on Software Engineering and Methodology, 34(2), Article 42. https://doi.org/10.1145/3697010
30. Ozturk, O., Ergul, M., Cagir, Y., Atay, A., Acun, K. C., Coskun, O., … Yuksel, I. (2025). Assessing ChatGPT-v4 for guideline-concordant inflammatory bowel disease care: Accuracy, completeness, and temporal drift. Journal of Clinical Medicine, 14(13), Article 4599. https://doi.org/10.3390/jcm14134599
31. Öner, S. K., Demirkiran, N. D., Canlı, E. A., & Bilir, A. (2026). Gonarthrosis Advisor versus ChatGPT-5: Quality and readability of artificial intelligence-generated patient education for knee osteoarthritis. Acta Orthopaedica et Traumatologica Turcica, 60(2). https://doi.org/10.5152/j.aott.2026.25618
32. Özer Aslan, İ., & Aslan, M. T. (2025). Benchmarking AI chatbots for maternal lactation support. Healthcare, 13(14), Article 1756. https://doi.org/10.3390/healthcare13141756
33. Pan, A., Musheyev, D., Bockelman, D., Loeb, S., & Kabarriti, A. E. (2023). Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncology, 9(10), 1437–1440. https://doi.org/10.1001/jamaoncol.2023.2947
34. Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37. https://doi.org/10.48550/arXiv.2404.13076
35. Qiu, Z., Jiang, A., Qi, C., Liu, Y., Lin, A., & Luo, P. (2025). Temporal evolution of large language models in oncology. Journal of Translational Medicine, 23(1), Article 1219. https://doi.org/10.1186/s12967-025-07227-2
36. Sargın, F., Alçı, M., Kaygusuz Aydemir, B., Şahin, S., Çaksen, H., & Güven, A. S. (2025). Evaluation of the readability and quality of online Turkish materials in the field of pediatric neurology. Turkish Archives of Pediatrics, 61(2), 147–151. https://doi.org/10.5152/TurkArchPediatr.2025.25337
37. Scaff, S. P. S., Reis, F. J. J., Ferreira, G. E., Jacob, M. F., & Saragiotto, B. T. (2025). Assessing the performance of AI chatbots in answering patients’ common questions about low back pain. Annals of the Rheumatic Diseases, 84(1), 143–149. https://doi.org/10.1136/ard-2024-226202
38. Shoemaker, S. J., Wolf, M. S., & Brach, C. (2014). Development of the Patient Education Materials Assessment Tool (PEMAT): A new measure of understandability and actionability for print and audiovisual patient information. Patient Education and Counseling, 96(3), 395–403. https://doi.org/10.1016/j.pec.2014.05.027
39. Şimşek, E., & Kurt, Ö. (2025). The impact of language differences on the readability, quality, and reliability of information provided by artificial intelligence chatbots regarding vital pulp therapy: A cross-sectional study. BMC Oral Health, 26(1), Article 134. https://doi.org/10.1186/s12903-025-07520-y
40. Wang, W., Zhou, Y., Fu, J., & Hu, K. (2025). Evaluating the performance of DeepSeek-R1 and DeepSeek-V3 versus OpenAI models in the Chinese national medical licensing examination: Cross-sectional comparative study. JMIR Medical Education, 11, Article e73469. https://doi.org/10.2196/73469
41. Wu, D. J. H., Nateghi Haredasht, F., Wu, D., Adelman, R. D., Chen, J. H., & Chen, C. Y. A. (2026). Automated evaluation of large language model response concordance with human specialist responses on physician-to-physician eConsult cases. Pacific Symposium on Biocomputing, 31, 372–387. https://doi.org/10.1142/9789819824755_0026
42. Wu, Z., Gan, W., Xue, Z., Ni, Z., Zheng, X., & Zhang, Y. (2024). Performance of ChatGPT on nursing licensure examinations in the United States and China: Cross-sectional study. JMIR Medical Education, 10, Article e52746. https://doi.org/10.2196/52746
43. Yao, Z., Duan, L., Xu, S., Chi, L., & Sheng, D. (2025). Performance of large language models in the non-English context: Qualitative study of models trained on different languages in Chinese medical examinations. JMIR Medical Informatics, 13, Article e69485. https://doi.org/10.2196/69485
44. Zaretsky, J., Kim, J. M., Baskharoun, S., Zhao, Y., Austrian, J., Aphinyanaphongs, Y., … Feldman, J. (2024). Generative artificial intelligence to transform inpatient discharge summaries to patient-friendly language and format. JAMA Network Open, 7(3), Article e240357. https://doi.org/10.1001/jamanetworkopen.2024.0357
45. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., … Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. https://doi.org/10.48550/arXiv.2306.05685
46. Zhou, M., Pan, Y., Zhang, Y., Song, X., & Zhou, Y. (2025). Evaluating AI-generated patient education materials for spinal surgeries: Comparative analysis of readability and DISCERN quality across ChatGPT and DeepSeek models. International Journal of Medical Informatics, 198, Article 105871. https://doi.org/10.1016/j.ijmedinf.2025.105871